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SUMMARY 


This  guide  provides  sufficient  background  and  technical  information  to  citable  a  measurement 
facilitator  to  use  the  Methodology  for  Generating  Efficiency  and  Effectiveness  Measures  (MGEEM) 
to  create  a  complete  productivity  measurement  system  for  any  organization.  Requisite  background 
information  includes  knowledge  of  the  systems  concept  so  that  the  organization  for  which  the 
MGEEM  is  to  be  applied  can  be  meaningfully  defined  in  terms  of  its  inputs,  outputs,  goals,  and 
interactions  with  its  environment.  According  to  the  systems  concept,  productivity  has  two 
components:  (a)  efficiency,  the  quantity  of  inputs  required  to  produce  a  given  level  of  outputs 
and  (b)  effectiveness,  the  extent  to  which  these  outputs  conform  to  mission  requirements. 

After  becoming  familiar  with  a  target  organization  and  defining  it  as  a  system,  a  measurement 
facilitator  develops  a  plan  for  implementing  the  MGEEM  which  includes  a  strategy  for 
strengthening  “forces  for"  and  minimizing  "forces  against"  the  measurement  process.  Within  the 
context  of  the  implementation  plan,  the  facilitator  begins  Phase  One  of  the  MGEEM  process  by 
convening  a  meeting  of  the  commander  (manager  or  supervisor)  and  his/her  immediate  subordinates. 
This  group  is  posed  with  the  question  "What  does  the  Air  Force  pay  this  organization  to  do?" 
Consensus  on  the  answers,  called  Key  Result  Areas  (KRAs),  is  developed  through  a  structured  group 
process.'  Phase  Two  involves  a  meeting  with  the  commander's  suborai nates  and  selected  members  of 
their  subordinates.  For  each  KRA,  the  facilitator  asks  what  the  commander  needs  to  know  to 
determine  how  well  the  organization  is  doing  on  this  aspect  of  the  organization's  productivity. 
Consensus  on  the  answers,  called  indicators,  is  again  developed  through  a  structured  group 
process.  Examples  of  KRAs  and  indicators  are  (a)  for  a  repair  shop,  quality  of  repair  and 
quality  control  Inspections  passed;  (b)  for  personnel,  timely  and  effective  staffing  of  positions 
and  percentage  of  positions  filled;  and  (c)  for  engineering,  customer  support  and  percentage  of 
work  orders  accomplished.  In  Phase  Three  of  the  MGEEM  process,  computerized  indicator  data  are 
periodically  reported  to  workers  and  managers  according  to  the  principles  of  feedback,  goal 
setting,  and  Incentives.  These  principles  have  to  do  with  issues  such  as  the  frequency  and 
objectivity  of  feedback;  the  acceptability,  difficulty,  and  feasibility  of  goals;  and  the 
connection  between  effort,  performance,  and  reward.  Suggestions  are  made  to  overcome  barriers  to 
successful  implementation. 
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PREFACE 


This  Is  one  of  two  final  technical  papers  from  Task  13,  Tecnnology  Transfer  Plan 
and  User  Manuals  for  the  Methodology  for  Generating  Efficiency  and  Effectiveness 
Measures  (MGEEM),  for  Contract  ho,  F3361 5-83-C-0030,  Contributlve  Research  in  Manpower 
and  Personnel  Technologies.  The  other  technical  paper  is  AFHRl-TP-86-26,  Methodol ogy 
for  Generating  Efficiency  and  Effectiveness  Measures  (MGEEM):  A  Gu^de  for  Commanders, 
Managers,  and  Supervisors.  Both  technical  papers  are  user  manuals  for  the  MGEEM.  The 
conceptualization  and  field  test  of  the  MGEEM  are  documented  in  AFHRl-TR-81 -9, 
Productivity  Measurement  Methods:  Classification,  Critique,  and  Implications  for  the 
Air  Force,  and  AFHRL-TP-84-54,  Field  Test  of  the  Methodology  for  Generating  Efficiency 
and  Effectiveness  Measures. 

The  authors  wish  to  express  their  appreciation  to  the  organizations  and  individuals 
who  participated  in  the  review  of  this  document.  In  particular,  we  thank  Col  Robert  M. 
DePhillipis,  Chief  of  the  Management  Consulting  Branch,  Leadership  and  Management 
Development  Center  ilMDC.'MC)  ana  Lt  Col  Arthur  R.  (Bob)  Dahms,  I !  1  and  Maj  Dwight  E. 
Beauchamp  in  the  Productivity  Office  of  the  Requirements  Division.  Directorate  of 
Manpower  and  Organization  (HQ  USAF/PMRE),  These  reviewers  made  substantial  inputs  to 
the  final  manuscript  which  enhance  its  relevance  for  Air  Force  users. 
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METHODOLOGY  FOR  GENERATING  EFFICIENCY 
AND  EFFECTIVENESS  MEASURES  (MGEEM): 

A  GUIOE  FOR  AIR  FORCE  MEASUREMENT  FACILITATORS 


1.  PRODUCTIVITY  IN  AIR  FORCE  ORGANIZATIONS 
I. I  What  Is  Productivity? 

What  does  productivity  mean?  Manpower  cuts?  More  for  less?  Peacetime  efficiency  at  the 
expense  of  wartime  readiness?  Or  could  it  mean  quality  products  and  services,  improved  military 
preparedness  for  the  tax  dollar  spent,  more  effective  organization,  higher  utilization  rates  for 
equipment,  higher  reenlistment  rates,  and  a  reduction  of  maintenance  costs  per  sortie? 

To  military  commanders,  productivity  may  have  any  or  all  of  these  connotations.  For  most 
commanders,  productivity  has  various  meanings,  depending  on  their  position,  background,  and 
management  philosophy.  However,  for  the  Air  Force  as  a  whole,  there  is  little  doubt  that  the 

concept  of  productivity  and  the  need  for  productivity  improvement  will  exist  for  the  foreseeable 

future.  .As  long  as  there  are  multiple  threats  to  the  nation’s  security  and  limited  resources  to 
deal  with  these  threats,  the  military  faces  the  problem  of  how  to  obtain  the  best  defense  for  the 

available  dollars  and  manpower.  This  is  what  productivity  is  all  about. 

As  a  measurement  facilitator,  you  are  most  likely  to  work  with  units  at  the  wing  level  and 
below.  Therefore,  you  need  to  think  about  what  productivity  means  at  these  organizational 
level?.  Before  discussing  the  meaning  of  productivity,  we  need  to  introduce  some  ideas  about 
organizations  as  systems.  This  will  already  be  familiar  to  many  Air  Force  managers  and  management 
consultants,  but  it  is  essential  to  an  understanding  of  the  Methodology  for  Generating  Efficiency 
and  Effectiveness  Measures  (MGEEM). 


1.2  Basic  System  Concepts 


A  system  Is  a  set  of  Interrelated  components  that  have  a  common  purpose  or  goal.  An  Air 
Force  organizational  system  could  be  a  wing,  a  squadron,  or  even  a  small  wc-k  team  within  a 
squadron.  Systems  can  be  small  or  large.  However,  they  all  have  a  set  of  interrelated 
components,  such  as  people,  materials,  equipment,  and  organizational  entities,  which  share  a 
common  mission. 

Systems  are  defined  in  terms  of  their  Inputs,  outputs,  goals,  and  interactions  with  their 
environment  across  system  boundaries.  Let  us  examine  these  concepts  for  a  military  organization, 
a  communfcatlons-navigatlon  (comn-nav)  branch  in  an  avionics  squadron.  This  branch  performs 
maintenance  on  the  on-board  communications  and  navigation  equipment  carried  by  aircraft  in  the 

wing. 


1.2.1  Inputs 

As  a  general  rule,  inputs  to  a  maintenance  organization  can  be  grouped  according  to  the 
categories  in  Table  1. 


I 


Table  1 .  Categories  of  Inputs 


Labor  -  Number  of  hours  worked 

Material  -  Number  of  pieces,  number  of  units,  dollar  value,  etc. 

Capital  -  Number  of  dollars  invested,  depreciated  value  of  assets,  etc. 
Energy  -  Number  of  kilowatt  hours  used,  number  of  gallons  of  fuel,  etc. 


For  the  comm-nav  branch,  labor  inputs  consist  of  the  total  number  of  assigned  personnel 
multiplied  by  the  number  of  duty  hours  in  the  period  under  consideration  (e.g.,  week,  month).  In 
certain  cases,  it  may  be  desirable  to  adjust  the  total  labor  input  for  such  factors  as  personnel 
on  temporary  duty  (TDY),  on  detail  to  other  units,  on  leave,  etc.  to  arrive  at  the  total  number 
of  available  hours.  Although  definitions  must  be  explicitly  agreed  upon,  labor  input  is 
typically  defined  in  terms  of  the  number  of  hours  spent  by  designated  personnel  during  a  given 
time  period.  In  most  military  organizations,  the  most  significant  input  is  labor. 

A  second  input  category  for  the  comm-nav  branch  is  material.  In  order  to  perform  its 

mission,  personnel  must  be  supplied  with  spare  parts,  expendable  items  (e.g.,  non-capi tali  zed 
tools  and  equipment),  office  supplies,  etc.  Quantifying  material  inputs  presents  a  number  of 
measurement  problems  because  the  categories  of  measurement  vary  from  item  to  item.  For  example, 
If  we  were  concerned  with  wire,  we  would  probably  quantify  it  in  terms  of  feet.  If  we  were 
concerned  with  pencils,  we  would  count  the  number  of  pencils  or  number  of  dozens.  For  avionics 
compofhents,  we  would  count  the  number  of  items.  Obviously,  we  cannot  aggregate  or  add  "apples 
and  Oranges"  unless  they  are  converted  to  a  common  scale  of  measurement.  Typically,  the  common 
scale  for  material  is  dollars.  Depending  on  the  accounting  practices  of  the  unit,  item  values 
can  be  expressed  in  terms  of  initial  cost  or  in  terms  of  replacement  cost. 

A  third  category  of  inputs  is  capital.  At  the  branch  level,  there  is  usually  very  little 
knowledge  of  capital  investment  program  opportunities  or  the  process  by  which  assets  are 

capitalized.  However,  the  unit  of  measurement  for  capital  assets,  such  as  buildings  and  major 
pieces  of  test  equipment,  is  units  or  dollars  of  depreciation.  In  wing  or  higher  level 
organizations,  capital  may  have  more  Importance.  Refer  to  Air  Force  Regulation  (AFR)  25-3,  Air 
Force  Productivity  Enhancement  Program  (PEP)  for  a  description  of  Air  Force  capital  investment 
programs. 

A  fourth  category  of  Inputs  is  usually  called  energy.  This  category  includes  the  electricity 
to  operate  the  facility,  the  fuel  to  power  vehicles  or  aircraft,  and  gas  or  oil  for  heating.  In 

the  case  of  the  comm-nav  branch,  energy  Is  not  a  significant  input.  Perhaps  the  branch  owns 

vehicles  and  can  affect  the  gallons  of  fuel  used.  It  can  also  affect  the  amount  of  energy  used 
to  heat  Its  facilities.  However,  energy  matters  are  typically  handled  by  a  centralized  resource 
management  agency.  When  a  single  energy  type  is  considered,  it  is  preferable  to  quantify  it  in 
terms  of  physical  units  such  as  gallons,  kilowatt  hours,  or  British  thermal  units.  However,  if 
aggregation  Is  an  Issue,  units  must  be  converted  to  dollars. 

1.2.2  Outputs 

A  second  major  systems  concept  is  outputs.  An  output  is  a  product  or  service  that  a  system 
produces  to  achieve  Its  purpose.  In  the  comm-nav  example,  outputs  are  pieces  of  equipment 
repaired  personnel  trained,  reports  submitted  to  squadron  headquarters,  etc.  Whatever  the 
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organization  produces  for  use  outside  its  own  organizati  onal  boundaries  is  an  output.  A 
distinction  is  frequently  made  between  "intermediate"  outputs  and  "final"  outputs.  Intermediate 
outputs  are  consumed  by  the  organization  itself  or  are  necessary  for  use  by  the  organization  to 
produce  final  outputs,  final  outputs  are  exported  across  the  organization's  boundaries  to  other 
organizational  systems. 


..2.3  System  Boundary 


The  boundary  of  a  system  separates  it  free,  its  environment.  Although  the  concept  of  boundary 
Is  "theoretical,"  it  has  practical  implications.  However,  defining  a  system  in  terms  of  what  it 
includes  is  not  always  easy.  For  instance,  in  defining  the  comm-nav  branch  ss  a  system,  where  do 
you  draw  the  line?  Is  an  individual  who  is  on  detail  to  the  squadron  for  ?  months  within  the 
system?  Are  the  vehicles  on  permanent  loan  to  the  branch  from  the  meter  pool  part  of  the 
comm-nav  system  or  the  motor  pool  system?  Questions  such  as  these  UvSt  be  answered  to 
unambiguously  define  the  system  under  investigation.  Once  such  questions  a.e  answered,  the 
system's  boundaries  can  be  drawn.  However,  it  should  be  clear  from  these  examples  that  boundary 
issues  deal  with  both  space  ( e. g - .  area  occupied  by  system  components),  ar.o  time  (e.g.,  the 
length  of  time  someone  is  away  from  the  organization  before  that  person  is  considered  no  longer 
within  the  system). 


1.2.4  Environment 


Once  a  system  is  defined  by  specifying  its  boundaries,  its  environment  must  also  be  defined. 
Anything  that  is  not  part  of  the  system  is  part  of  its  environment.  Systems  survive  through 
transactions  with  their  environment.  Personnel  resources,  supplies  and  equipment,  and  energy  to 
operate  the  system  flow  from  the  environment.  Relevant  organi zati ons  in  the  environment  of  the 
comm-nav  branch  include  the  maintenance  directorate,  avionics  squadron  headquarters ,  other 
branches  in  the  avionics  squadron,  supply  organi zati  ons ,  and  flignt  operations  (since  many  of  the 
branch's  mission  goals  flow  from  equipment  writeups  made  by  flight  crews).  Maintaining  effective 
boundary  relationships  is  a  critical  aspect  of  a  system's  operation. 


1.2.5  System  Diagram 


These  basic  system  concepts  may  be  depicted  in  the  fern,  of  a  diagram,  as  shown  in  figure  1. 


ENVIRONMENT 


T 
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. 1  I 

[  I 

INPUTS  . >  OUTPUTS 

L . BOUNDARY . J 

.  ENVIRONMENT  . 

Figure  1.  System  Diagram. 

As  Figure  1  shows,  a  system  is  defined  by  its  boundary,  which  separates  it  from  its 
envi ronmenc.  In  an  open  system,  such  as  an  organizational  system,  inputs  are  derived  from  tne 
environment.  The  system  uses  these  inputs  to  produce  outputs  which  are  exchanged  with  the 
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environment.  As  this  very  simple  analysis  suggests,  in  organizational  system  must  manage  the 
process  of  converting  Inputs  to  outputs,  but  must  also  manage  Its  boundary  relationships  to 
assure  a  continuing  flow  of  Inputs.  In  most  organizational  systems,  the  flow  of  Inputs  Is 
dependent  on  the  extent  to  which  system  outputs  conform  to  the  requirements  of  the  environment 
(e.g.,  customers)  with  respect  to  quantity,  quality,  price/cost,  etc. 


i .3  Productivity  -  A  Systems  Concept 

Now  that  we  have  discussed  the  basic  components  of  an  organizational  system,  we  are  In  a 
position  to  define  productivity.  As  applied  to  Air  Force  organizations,  productivity  has  two 
components:  efficiency  and  effectiveness.  Efficiency  Is  concerned  with  the  quantity  of  inputs 
required  to  produce  a  given  level  of  outputs.  Effectiveness  refers  to  the  extent  to  which  the 
outputs  conform  to  mission  requl rements.  These  definitions  are  expressed  in  AFR  25-3,  which 
defines  productivity  as: 

...the  measure  of  an  organi zatlon' s  performance.  It's  not  only  "efficiency"  (the 
ratio  of  inputs  to  outputs),  but  also  "effectiveness”  (to  what  extent  the  output 
satisfies  mission  objectives)  (AFR  25-3,  1982,  p.  2). 


2.  LAYING  THE  FOUNDATION 
2.1  Selecting  a  Measurement  Facilitator 

The  MGEEM  process  (s  conducted  by  a  measurement  facilitator  who  Is  usually  someone  external 
to  the  organization,  although  not  necessarily  so.  The  measurement  facilitator  should  possess  a 
working  knowledge  of  and  demonstrated  skill  In  group  facilitation  (e.g.,  conducting  meetings, 
facilitating  group  discussion,  interviewing,  bringing  about  consensus,  and  listening).  Although 
specific  facilitation  skills  unique  to  the  MGEEM  are  covered  in  this  manual,  no  attempt  is  made 
to  provide  a  full  treatment  of  the  range  of  skills  required.  Measurement  facilitators  lacking 
group  facilitation  experience  should  obtain  guidance  from  a  "process"  coach.  In  Air  Force 
organizations,  such  coaches  might  be  quality  circles  facilitators  or  human  relations  trainers,  or 
persons  working  In  headquarters  consulting  organizations.  In  addition,  the  facilitator  should 
understand  the  basics  of  organizational  performance  measurement.  AFR  25-3  is  a  helpful  guide  In 
this  respect,  especially  Its  General  Information  and  Glossary  sections. 


2.2  Organizational  Familiarization 

Once  selected,  the  measurement  facilitator  must  become  familiar  with  the  target 
organization.  This  can  be  accomplished  through  (a)  discussions  with  the  commander  (manager  or 
supervisor),  (b)  a  review  of  existing  In-house  documentation,  and  (c)  a  tour  of  the 
organization's  facilities  and  work  sites. 


2.2.1  Discussions  with  the  Commander 


Ar  Initial  meeting  with  the  target  unit's  commander  marks  the  formal  beginning  of  the  MGEEM 
process.  If  the  commander  has  Initiated  the  process  and  selected  the  measurement  facilitator, 
then  this  meeting  should  focus  on  the  commander's  purpose  In  Initiating  the  measurement  activity 
and  on  helping  the  measurement  facilitator  become  familiar  with  the  organization.  However,  In 
‘.he  event  that  the  commander  has  been  directed  by  higher  headquarters  to  begin  the  measurement 
process  -  not  the  Ideal  situation  -  the  Initial  meeting  will  be  of  a  different  nature.  In  that 


4 


case,  the  measurement  facilitator  should  devote  considerable  time  and  attention  tc  establishing 
rapport  with  the  commander  to  assure  him/her  that  the  purpose  of  the  MGEEM  process  Is  to  develop 
a  measurement  tool  that  will  improve  the  management  of  the  unit. 

Based  on  the  assumption  that  the  commander  is  voluntarily  engaging  in  the  MGtEM  process,  the 
facilitator  can  use  the  structured  interview  form  in  Appendix  A  to  assist  In  the  Initial 
meeting.  In  this  meeting,  it  is  likely  that  the  commander  will  expect  to  receive  an  overview 
briefing  on  the  MGEEM  implementation  process.  Therefore,  the  facilitator  should  be  prepared  to 
discuss  the  major  steps  in  the  process  and  provide  a  timetable  for  their  implementation.  The 
Implementation  planning  process  is  discussed  in  Section  2.3.3. 


2.2.2  Review  of  Existing  In-House  Documentation 

In  most  Air  Force  functional  areas,  the  measurement  facilitator  can  become  sufficiently 
familiar  with  the  target  organization  and  its  functions  by  reviewing  written  material  relative  to 
the  organization.  Examples  of  documentation  that  can  be  included  in  this  review  are: 

1.  Work  center  descriptions, 

2.  Task  inventories  containing  detailed  task  listings  of  the  Air  Force  Specialty  Codes 
Invol ved; 

3.  Organization  ,nd  work  center  organizational  charts; 

4.  Manpower  studies  conducted  for  the  organization  or  for  the  functional  specialty; 

5.  Regulatl ons/di recti ves  and  other  official  documents  that  govern  the  work  of  the  target 
organization; 

6.  Commander's  briefing  packages  used  to  give  visitors  an  overview  of  the  organization. 


2.2.3  Site  Visit 


Another  source  of  information  about  the  target  organization  Is  a  site  visit.  Prior  to 
conducting  such  a  visit,  the  facilitator  and  commander  should  decide  how  the  facilitator  Is  to  be 
Introduced  to  unit  personnel.  If  the  visit  Is  to  be  conducted  prior  to  the  Implementation  of  the 
measurement  communications  plan,  then  the  facilitator  should  be  introduced  as  a  visitor,  if  unit 
personnel  have  been  fully  Informed  about  the  purpose  and  events  associated  with  the  MGEEM 
Implementation  process,  the  facilitator  should  be  Identified  as  being  associated  with  the 
process.  The  issue  is  that  the  facilitator's  visit  through  the  organization  Is  not  the 
appropriate  time  to  communicate  to  unit  personnel  what  the  measurement  process  Is  about. 

During  the  site  visit,  the  facilitator's  primary  concern  is  to  develop  a  visual  picture  and 
to  "learn  the  language"  of  the  target  organization.  The  facilitator  should  focus  on  identifying 
the  unit's  major  products  and  services  and  developing  a  general  understanding  of  the  work  flow 
Involved.  Although  It  is  possible  for  the  facilitator  to  do  an  adequate  Job  of  conducting  the 
MGEEM  process  without  having  seen  the  work  site,  there  are  advantages  to  such  a  visit.  One 
advantage  Is  the  opportunity  to  establish  rapport  with  members  of  the  organization.  if  the 
facilitator  cares  enojgh  to  see  how  the  organization  works,  members  will  tend  to  accord  him/her 
greater  respect.  Another  advantage  is  that  through  seeing  the  work  site,  the  facilitator  will 
communicate  more  effectively  In  the  group  sessions  to  follow.  The  facilitator  will  better 
understand  participants'  comments  without  needing  to  seek  clarification. 


2.2.4  Organisational  Diagram 

The  facilitator  should  depict  the  results  of  the  organizational  f ami  1 iari zation  process  In 
the  form  of  an  organizational  diagram.  In  essence,  this  is  the  systems  diagram  shown  In  Figure  1 
loaded  with  characteristics  of  the  specific  target  organization.  A  resulting  sample 
organizational  diagram  for  a  base  administration  division  Is  shown  In  Figure  2. 


2.3  Planning  the  Measurement  Process 


2.3.1  Defining  Goals  of  the  Measurement  Activity 


The  first  step  In  developing  a  measurement  plan  Involves  working  with  the  commander  to  define 
the  goals  of  the  productivity  measurement  process.  There  are  many  reasons  why  organizations 
decide  to  generate  measures  of  their  productivity.  In  general,  these  motives  may  be  grouped 
under  (a)  organl zatlonal  control  and  (b)  organizational  Improvement.  The  motive  of 
organizational  control  Is  that  organizations  want  to  generate  measures  to  satisfy  the  needs  of 
someone  else  (e.g.,  higher  headquarters  or  a  control  agency,  such  as  manpower  or  budget).  The 
motive  of  measurement  for  orgariz:Mr,'a1  Improvement  has  a  different  focus,  in  that  the 
commanders  themselves  wish  to  create  measures  as  guides  to  take  corrective  action  to  Improve 
organizational  performance.  Thus,  control  -oriented  measurement  Is  used  by  organizations  to 
Justify  themselves  to  organizations  in  their  environments.  Improvement-oriented  measurement 
provides  information  to  enable  organizations  to  better  manage  themselves.  Control -oriented 
measurement  is  commonplace  in  the  Air  Force,  but  improvement-oriented  measurement  occurs  less 
frequently.  Since  the  MGEEM  is  primarily  an  improvement-oriented  approach,  this  distinction 
needs  to  be  recognized. 

A  basic  purpose  of  developing  Improvement-oriented  measurement  is  that  It  provides  an 
Organization  with  a  considerable  degree  of  self-control .  Therefore,  the  relevant  measures  are 
not  those  which  fall  under  the  control  of  external  agencies.  For  the  purposes  of  MGEEM,  the  most 
relevant  measures  are  those  that  are  controllable  from  the  point  of  v'ew  of  the  unit  commander. 

Measurement  should  help  the  commander  periodically  identify  and  track  those  aspects  of 
efficiency  and  effectiveness  over  which  he/she  has  control.  Through  measurement,  the  unit  can 
assess  Its  current  performance  and  take  corrective  action  as  required.  The  facilitator  should 
work  with  the  commander  to  promote  Improvement  orientation  as  the  purpose  of  measurement. 
Measures  being  so  developed  are  for  the  unit  Itself,  not  for  some  outside  control  agency  or 
higher  headquarters. 


2.3.2  Identifying  Forces  For  and  Against  Success  of  the  Measurement  Activity 

At  this  stage  of  the  process,  the  facilitator  should  work  with  the  commander  to  identify  the 
driving  and  restraining  forces  associated  with  Implementation  of  the  MGEEM  process. 
Identification  of  these  forces  can  be  portrayed  as  a  diagram  (Figure  3).  At  the  discretion  of 
the  unit  commander.  It  may  be  helpful  to  Involve  other  managers,  supervisors,  and  Individual 
workers  In  this  effort. 


2.3.3  Developing  an  Implementation  Plan 

Once  the  "forces  for  and  against"  analysis  has  been  completed,  the  result  should  serve  to 
guide  the  development  of  an  Implementation  plan.  The  strategy  to  follow  in  developing  an 
Implementation  plan  Is  to  consider  options  for  maximizing  or  strengthening  the  forces  for  and 
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minimizing  the  forces  against  successful  implementation.  A  diagram  such  as  that  shown  In  Figure 
3  can  guide  this  process. 

For  example,  consider  the  first  item  under  forces  For  in  Figure  3,  "Commander  Support." 
Commander  Support  is  a  force  for  MGEEM  implementation,  but  only  if  It  is  communicated  clearly  and 
strongly  to  all  members  of  the  organization.  Therefore,  an  important  step  In  the  Implementation 
plan  is  to  communicate  the  support  of  the  commander  to  organization  members.  A  similar  process 
can  be  carried  out  for  other  Forces  For  items. 

Goal  -  Successful  Implementation  of  MGEEM 


Forces  for 

Forces  against 

Commander  Support  — - — 

—  - > 

< . 

Desire  to  Improve  - 

< . 

-  Heavy  Workload 

Noncommissioned  Officer 

(NC0)  Support  - 

—  - > 

<- . 

Personnel  Desire 

<- . 

to  Have  a  voice 

of  Productivity 

<- . 

Freedom 

Figure  3.  Analysis  of  Forces  For  and  Forces  Against  Successful  Implementation 
of  MGEEM. 

Forces  Against  can  be  addressed  in  the  Implementation  plan  by  (a)  acknowledging  them,  (b) 
reducing  them,  (c)  showing  that  there  Is  no  substantial  reason  for  concern,  or  (d)  removing  the 
reason  for  the  fear.  For  example.  Fear  of  More  Paperwork  might  be  removed  or  reduced  by  a 
notation  In  the  Implementation  plan  of  the  need  to  acquire  a  persona)  computer,  or  Fear  of  Loss 
of  Freedom  might  be  addressed  in  the  orientation  process  prior  to  measurement  development.  A 
sample  Implementation  plan  format  is  presented  in  Appendix  8. 


2.4  Communicating  the  Implementation  Plan 


After  the  measurement  plan  Is  developed,  It  should  be  communicated  over  a  short  period  of 
time  to  all  members  of  the  organization  from  the  top  down.  Communication  should  be  In  the  form 
of  both  face-to-face  and  written  communication.  The  sequence  of  communication  activities  for  a 
branch  might  be  as  fellows. 

Monday,  0800,  Branch  Chief's  Staff  Meeting.  The  branch  chief  Introduces  the  facilitator  and 
discusses  the  purpose  of  the  MGEEM  measurement  activity  within  the  branch.  The  facilitator 
discusses  the  teps  Involved  In  Implementation  and  proposes  a  timetable.  Section  chiefs  are 
asked  for  comments,  and  schedule  conflicts  are  resolved.  The  branch  and  section  chiefs  agree  on 
the  appropriate  time  for  the  facilitator  to  visit  the  various  sections  to  present  the  measurement 
plan. 


Monday,  0900,  Facilitator  and  Branch  Chief  Finalize  Memorandum.  A  memorandum  Is  prepared  for 
later  distribution  to  personnel,  summarizing  the  highlights  of  the  Implementation  schedule.  The 
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memorand'.'':  should  focus  on  the  purpose  of  the  meas urement  activity,  the  schedule-,  and  now  members 
of  the  organization  will  oe  affected. 


Monday,  1300  -  1400,  workplace  Keelings.  The  facilitator  and  branch  chief  go  to  each  section 
and  hold  10-  to  15-minute  meetings  with  section  members.  Ouring  these  meetings,  mergers  hear  the 
branch  and  section  chiefs  express  support  for  the  MGEEM  measurement  process.  Members  hear  from 
the  facilitator  a  general  outline  of  the  steps  involved  in  implementation  and  what  will  be 
expected  from  each  member  of  the  organization.  Each  member  of  the  section  receives  a  copy  of  the 
previously  developed  memorandum,  which  summarizes  the  information  presentee.  Members’  questions 
are  answered. 


2.5  Pitfalls  to  Avoid 


At  this  stage  of  tne  process,  there  are  some  mayor  rules -of- thumb  tc  observe  in  order  to 
avoid  some  common  pitfalls: 

1.  Avoid  the  perception  trat  the  measurement  effort  is  being  forced  on  the  organization 
and  does  not  have  the  support  of  branch  and/or  section  chiefs.  Tne  main  purpose  of  the 
communications  effort  is  to  prevent  such  misperceptions.  Visible,  sincere  management  support  is 
essential  to  the  success  of  the  MGEEM  process. 

2.  Avoid  tne  perception  that  the  process  is  designed  to  "tighten  the  screws"  on  members 
of  the  organization.  Improvement,  not  control ,  should  bo  the  focus.  All  communications  should 
focus  on  these  points:  «e  (members  of  the  organization)  are  doing  this  for  our  own  benefit,  to 
give  us  a  score  card  on  our  performance,  to  provide  a  sense  of  accomplishment,  to  give  us 
feedback,  and  to  help  us  identify  areas  where  w e  can  improve  performance  in  the  sense  of  working 
more  efficiently,  not  necessarily  harder. 

3.  Recognize  in  the  p'anning  process  the  legitimate  organizational  t3rriers  to 
implementation  of  the  measurement  effort.  For  instance,  it  is  a  mistake  not  tc  recogni  „• 
expected  strains  on  resources,  such  as  large  numbers  of  people  on  TCY  and  on  exercise,  cr  to  fail 
to  pay  enough  attention  to  the  fears  of  organization  members.  Implementation  must  include  a 
sincere  effort  to  give  people  a  chance  to  voice  their  fears  and  concerns  aru  nave  those  concerns 
answered  honestly.  However,  implementation  should  not  be  allowed  to  drag  or  until  each  minor 
issue  is  resolved. 

4.  Ensure  that  the  target  organization  has  been  clearly  defined.  To  avoid 
misunderstandings.  the  facilitator  must  have  a  clear  conception,  displayed  in  an  organization 
diagram,  of  wnat  organizational  units  and  functions  arc  within  and  beyond  the  scope  of  the 
measurement  activity.  This  understanding  must  be  shared  with  and  agreed  to  by  the  commander  and 
management  of  the  target  organf zation. 


3.  DECIDING  WHAT  TO  MEASURE  IN  AIR  FORCE  ORGANIZATIONS 
3.1  Productivity  in  Air  Force  Organizations 


One  of  the  facilitator's  primary  responsibilities  is  to  help  the  com-ander  of  the  target 
organization  communicate  the  philosophy  of  productivity  improvement  to  organizational  members. 
This  requires  two  sets  of  competencies.  First,  the  facilitator  must  have  an  understanding  of  the 
meaning  of  productivity  improvement  as  it  relates  to  other  aspects  of  an  organi zation '  s 
functioning.  Second,  the  facilitator  must  be  able  to  communicate  this  understanding  meaningfully 
to  commanders,  officers,  enlisted  personnel,  and  civilian  employees.  Tre  facilitator  must 
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understand  and  communicate  the  meaning  of  productivity  to  these  members  whether  they  be  in 
operational  units  (e.g.,  fighter  squadrons,  jet  engine  maintenance  branches,  and  military  airlift 
wings)  or  in  support  units  (e.g.,  administration  divisions,  weather  detachments,  and  base 
contracting  offices).  This  section  provides  the  basic  concepts  of  productivity  for  the 
facilitator  and  suggests  a  means  of  communicating  these  ideas  to  target  organization  members. 

Air  Force  Regulation  25-3.  AFR  25-3  contains  useful  information  for  the  facilitator  on  the 
application  of  productivity  concepts  In  the  Air  Force  environment.  Of  particular  use  in  the 
MGEEM  process  are  the  General  information  and  Glossary  sections. 

Productivity  and  Readiness.  One  commonly  held  notion  among  Air  Force  commanders  is  that  the 
concept  of  productivity  Implies  achieving  peacetime  resource  savings  at  the  expense  of  wartime 
readiness.  This  Issue  Is  addressed  In  AFR  25-3,  which  notes  that  productivity  enhancement 
focuses  on  functions  that  are  essentially  the  same  in  peacetime  and  wartime  (e.g.,  launching  and 
recovering  aircraft,  repairing  components,  delivering  fuel,  and  procuring  and  distributing 

supplies). 

Discovering  and  Implementing  improved  ways  of  performing  these  peacetime  functions  will 

both  enhance  their  effectiveness  In  wartime  and  free  resources  for  use  in  Improving 

readiness  In  other  areas.  (AFR  25-3,  1982,  p.  2). 

King  commanders  Involved  in  a  field  test  of  the  MGEEM  (Tuttle,  Wilkinson,  4  Matthews,  1985) 
observed  that  In  the  Air  Force,  resource  constraints  are  a  way  of  life,  whether  In  peacetime  or 
wartime.  Therefore,  commanders  at  all  levels  must  be  concerned  with  efficiency  as  well  as 

effectiveness.  In  discussing  productivity  and  readiness,  however,  the  issue  is  not  one  of 
tradeoffs  between  the  two.  The  Issue  Is  assuring  the  maximum  level  of  readiness  for  a  given 

level  of  resources.  Stated  In  this  manner,  Improved  productivity  Is  clearly  consistent  with 

improved  levels  of  readiness. 

Productivity  and  Morale.  There  Is  often  a  tendency  to  think  that  productivity  Improvement  Is 
obtained  at  the  expense  of  morale.  Experience  suggests  that  while  this  tradeoff  sometimes 
occurs,  It  Is  more  frequently  the  case  that  organizations  which  perform  well  also  have  high 
morale.  High  performance,  high  productivity,  and  high  morale  are  compl emcntary ,  not  competing, 
aspects  of  organizational  performance.  The  key  to  this  linkage  lies  In  the  following  assumptions 
about  people  in  organizations: 

1.  Most  people  want  to  do  a  good  job. 

2.  Most  people  want  to  feel  that  they  are  a  part  of  an  organization  that  is  worthwhile  and 
is  acccmpl 1 shl ng  something. 

3.  Most  people  have  needs  for  security,  self-esteem,  social  acceptance,  recognition,  etc., 
which  they  attempt  to  satisfy  in  part  through  their  work  lives. 

4.  There  Is  an  overlapping  relationship  between  organizational  and  individual  goals. 

5.  Leadership  and  management  provide  the  linkage  between  the  satisfaction  of  Individual  and 
organl rational  goals. 

Thus,  a  well-managed  organization  pays  attention  to  the  needs  of  both  the  individual  and  the 
organization.  Such  organlzati ons  generally  have  high  productivity  and  high  morale. 
Organizations  that  are  geared  only  toward  the  goals  of  the  organization  may  show  high 
productivity  for  the  short  term,  but  morale  will  invariably  be  low.  Eventually,  this  form  of 
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management  leads  tc  low  produc  ti  vi  ty ,  unless  managers  have  a  nigh  degree  of  coercive  power  over 
the  members  of  the  organization. 

Productivity  in  Different  Organizational  Environments:  A  Frame  of  Reference.  The  systems 
view  of  organi za ti ons  presented  in  Chapter  1  provides  a  framework  for  analyzing  productivity  in 
Air  Force  organizations.  This  framework  can  be  expanded  to  consider  the  relative  importance  of 
efficiency  and  effectiveness  in  different  types  of  Air  Force  organizations. 

For  this  purpose,  there  are  three  types  of  systems:  (a)  direct  outcome,  (b)  ind  rect 
outcome,  and  (c)  unknown  outcome.  In  a  direct  outcome  system,  depicted  in  Figure  4,  oufput 
virtually  assures  outcome.  Although  there  is  some  choice  about  the  form,  design,  or  means  of 
presenting  output,  the  range  of  output  variability  is  relatively  low.  Examples  of  direct  outcome 
systems  in  the  Air  Force  Include  dining  halls,  reprographics  shops,  mail  handling  operations, 
supply  rooms,  and  maintenance  organizations.  In  suen  systems,  the  consistency  of  outputs 
virtually  assures  that  the  desired  outcomes  will  occur.  As  a  result,  the  major  considerations  on 
the  part  of  the  producing  system  are  efficiency  and  quality.  Such  organi zations  are  concerned 
with  efficiency  because  they  must  use  resources  skillfully  in  order  tc  produce  a  sufficient 
quantity  of  outputs  in  the  appropriate  time  period.  They  are  concerned  with  quality  because  a 
principal  outcome,  customer  sa ti sfac ti on ,  is  directly  related  to  the  extent  to  which  output 
corresponds  to  customer  requirements,  in  this  type  of  system,  effectiveness  (i.e.,  choice  of  the 
appropriate  output  tc  accomplish  the  mission)  i<  less  of  a  concern  because  it  is  less 
controllable  by  the  organi zation. 


Input  - >  Output  - >  Outcome 


Figure  4.  Oirect  Outcoeie  System. 

Figure  5  depicts  an  Indirect  outcome  system,  in  which  the  range  of  variability  of  outputs  is 
greater.  The  system  must  choose  from  among  a  range  of  possible  outputs  those  which  will  produce 
the  most  favorable  outcomes.  Only  after  outputs  are  clearly  specified  should  the  system  focus  or, 
improving  the  efficiency  with  which  outputs  are  produced.  The  first  concern  of  such  an  indirect 
outcome  system  is  whether  chosen  outputs  are  the  correct  ones.  Examples  of  such  systems  in  the 
Air  Force  are  weather  detachments,  administration  communl cation  branches,  inspector  general 
teams,  management  engineering  teams,  and  military  airlift  squadrons.  In  each  of  these 
organizations,  the  commander  chooses  from  a  broad  range  of  possible  outputs  those  which  are  most 
appropriate  in  terms  of  mission  accomplishment.  Effectiveness,  tne  extent  to  which  the  outputs 
lead  to  desired  outcomes,  is  a  major  criterion  of  organizational  performance.  Efficiency  is  also 
Important,  but  only  after  the  organization  has  determined  that  it  is  producing  the  "right1' 
outputs. 


Input  - -->  Output  T  Cutccx- 


Figure  ?.  Indirect  Outcome  System. 
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Figure  6  presents  a  third  type  of  system,  the  unknown  outcome  system.  Many  combat 
Organizations  are  in  this  category.  An  unknown  outcome  system  is  one  that  prepares  to  perform 
its  mission  but  hopefully  never  is  called  on  to  actually  do  what  it  prepares  to  do. 


Input  - >  Output  Surrogate  Outcome 

Outcome 


Figure  6.  Unknown  Outcome  System. 

Air  Force  organizations  which  are  unknown  outcome  systems  Include  tactical  tighter  squadrons, 
bomber  squadrons,  and  even  flying  training  squadrons  that  teach  combat  maneuvers.  Primary  flight 
training  and  instrument  flight  training  organizations  are  indirect  outcome  systems  since  students 
have  the  opportunity  to  determine  if  the  outcomes  are  achieved  (e.g.,  basic  flight  maneuvers  and 
sorties  under  instrument  flight  rules).  In  an  unknown  outcome  system,  effectiveness,  measured 
using  surrogate  outcomes  (e.g.,  simulated  battle  conditions),  and  innovation  are  more  relevant 
criteria  than  efficiency.  Too  often  productivity  has  been  presented  in  ways  that  suggest  ar 
attempt  to  convert  unknown  outcome  systems  Into  direct  outcome  systems.  Clearly,  the  relevant 
criteria  must  differ  as  a  function  of  the  type  of  system  under  study. 

An  Important  conclusion  from  this  discussion  is  that  the  facilitator  must  understand  which  of 
the  three  types  of  systems  the  target  organization  most  closely  resembles.  The  facilitator 
should  then  guide  the  organization  toward  the  most  appropriate  mix  of  criteria  for  that  type  of 
system. 


] 
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3.2  Forming  the  Measurement  Development  Teams  (HDTsj 

Basic  Considerations.  Because  it  is  a  participative  methodology,  the  MGEEM  is  only  as  effective 
as  Its  participants.  Therefore,  the  facilitator  must  work  with  the  commander  of  the  target 
organization  to  assure  that  measurement  development  team  (MOT)  members  am  carefully  selectee. 
In  selecting  members  of  these  teams,  the  facilitator  should  be  ccncei'"ed  about: 

1.  Participant  Knowledge.  Do  prospective  team  members  'ave  sufficient  knowledge  of  the 
organization  and  Its  primary  work  processes  to  contribute  ideas  aid  maxe  neanlr.gful  judgments? 
Such  knowledge  among  members  is  usually  assured  by  obtaining  experienced  individuals  fron.  a  range 
of  grade  levels. 

2.  Participant  Communication  -kills.  Because  the  MGEtM  is  a  verbal  process,  the 
communication  skills  of  participants  are  important.  Not  only  must  participants  have  knowledge, 
they  must  also  be  able  to  communicate  tiieir  kncwleayt.  Or  t*~e  other  hand,  inexperienced 
Individuals  who  are  verbal  tend  to  create  problems  for  the  process.  Such  individuals,  unless 
carefully  managed  by  the  faciHtator,  can  pull  the  process  in  i  nappropri  ate  directions.  In 
addition  to  these  concorrs,  the  facil1tator  should  stress  to  the  commander  and  senior 
noncommissioned  officer  (NCC)  the  reed  lO  have  as  MOT  members  a  cross-section  of  the  organization 
and  to  Include  people  who  will  "speak  the!'  minds"  in  constructive  ways.  Management  should  have 
the  final  word  In  selecting  participants  because  it  is  management,  and  not  the  facilitator,  who 
must  Hve  with  the  results. 

3.  Posi tion/Influence  in  the  Orgarl zatlon.  A  basic  assumption  underlying  the  MGEEN  is  that 
participation  In  the  development  of  a  solution  will  increase  the  acceptability  of  that  solution. 
Therefore,  the  facilitator  should  seek  to  ensure,  ail  else  being  equal,  that  key  opi ni on-1 eaders 
In  the  organization  are  included  on  the  MDTs.  individuals  may  be  opi ni on-1 eaders  by  virtue  of 
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formal  or  informal  power.  It  is  important  tc  identify  these  key  individuals  and  consider  then' 
for  participation.  A  key  opinion-leader  who  is  left  out  can  jeopardize  acceptance  of  the 
resulting  MGEEM  products. 

4.  Comprehensiveness.  Tre  products  from  the  .“GEEK  will  be  best  if  MUl  members  have  the 
widest  possible  information  base.  Therefore,  the  best  situation  is  to  assure  representation  from 
multiple  points  of  view  (e.g.,  all  facets  of  the  target  organization,  next  higher  level 
organization,  and  significant  "customer”  organizations). 

One  MOT  or  Two?  In  most  cases,  it  is  desirable  to  form  two  MOTs  to  broaden  participation.  A 
management  MOT,  called  Team  A,  defines  the  broad  measurable  facets  of  the  unit's  mission,  laoeled 
Key  Result  Areas  (KRAs).  A  worker-level  MOT,  called  Team  B,  develops  the  specific  indicators  or 
measures  for  each  <RA.  There  are  exceptions  to  this  guideline.  If  the  organization  is  small 
(e.g.,  10  -  15  people),  then  forming  two  MDTs  is  probably  not  feasible.  If  the  organization  is 
comprised  of  professionals  (e.g.,  researchers,  physicians,  scientists,  engineers,  or  social 
workers),  who  are  accustomed  to  operating  in  a  participative  manner,  thc-y  may  be  uncomfortable 
with  having  only  managers  involved  in  developing  KRAs.  In  this  case,  one  MCI  comprised  of  botn 
managers  and  non-managers  should  be  used  to  develop  both  kRA's  and  indicators. 

Guidelines  for  Team  Formation.  In  determining  MOT  membership  for  a  typical  Air  Force 
organization,  you  must  first  (a)  identify  the  target  organization  (e.g.,  squadron),  (b)  identify 
the  next  highest  level  of  organization  (e.g.,  wing),  and  then  (c)  identify  the  next  two  levels 
below  the  target  organization  (e.g.,  branch  and  section),  as  appropriate.  Then  follow  t tie  steps 
outlined  in  Table  2. 


Table  2.  Guide  to  Determining  MOT  Membership 


Steps 

— 

Example 

1.  List  all  levels  in 

1. 

W1  ng 

nierarchica1  order;  identify  target 

2. 

Squadron* 

organization. 

3. 

Branch 

4. 

Section 

5. 

Workers 

?.  If  there  are  two  organizational  levels 

1. 

Wing 

below  the  target  organization  then: 

2. 

Squadron* 

3* 

Branch 

a.  Team  A  consists  of  levels  1,  2,  and  3 

4. 

Section 

b.  Team  8  consists  of  levels  3  and  4 

5# 

Workers 

3.  If  there  it  only  one  level  of  organization 

i. 

Squadron 

below  the  target  organization,  then: 

2. 

Branch3 

3. 

Section 

a.  Team  A  consists  of  levels  5,  2,  and  3, 
and  "customer"  representatives,  if 
possible 

b.  Team  8  consists  of  levels  2  and  3 

4. 

Workers 

4.  If  more  than  two  organizational  levels 

1. 

Wing 

can  be  defined  but  the  next  level*1  below 

Z. 

Squadron 

the  target  organization  is  a  worker-level 

3. 

Branch 

organization,  then; 

4. 

Section* 

a.  Team  A  consists  of  levels  1,  2,  ard  3 

b.  Team  8  consists  of  levels  3  and  4 

5. 

Workers 

S.  If  only  two  organizational  levels  can 

1. 

Branch 

be  defined,  then  Teams  A  and  B  are 

Z. 

Section* 

the  same  group  and  Include  representation 
from  the  level  above  and  the  worker  level. 

3. 

workers 

^Level  refers  to  the  lowest  level  with  a  manager/supervi sor. 
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3. 3  Developing  Key  Result  Areas  (KRAs) 


Key  Result  Areas  (nRAs)  arc  the  measurable  facets  of  an  organization's  mission,  from  the 
viewpoint  of  the  external  environment,  they  are  the  reasons  that  the  organization  exists.  In 
other  words,  higher-level  organ! nations,  customers,  and  others  have  expectation,  that  the  target 
organization  will  produce  certain  products  or  services  or  will  maintain  a  state  of  readiness  to 
perform  certain  activities.  Stated  in  measurable  terms,  these  expectations  are  the  KRAs  for  an 
organization,  Examples  of  KRAs  developed  for  a  range  of  Air  Force  organizations  are  presented  'in 
Appendix  C.  The  following  section  presents  the  steps  involved  in  developing  KRAs  and  offers 
suggestions  to  the  facilitator  as  to  how  to  implement  these  steps. 


3.3.1  Room  Selection  and  Setup 


A  key  factor  in  developing  KRAs  is  the  ability  of  tie  measurement  facilitator  to  function 
effectively  in  the  physical  space  arrangement.  The  room  should  be  well  ventilated,  with  adequate 
lighting  for  people  to  work  effectively  and  comfortably.  To  ensure  sufficient  wall  space  for 
hanging  chart  paper,  there  should  be  at  least  20  linear  feet  of  wall  space  free  of  pictures, 
windows,  doors,  and  other  obstructions.  The  room  si2e  should  be  sufficient  to  accommodate  a  long 
conference  table  or  L-shaped  arrangement  of  tables  which  faces  a  long  wall.  In  addition  to  a 
conference  table,  the  setup  should  include  at  least  one  easel  with  chart  paper  pad,  as  well  as 
markers,  masking  tape,  and  index  cards.  Having  markers  of  three  different  colors  is  helpful  for 
editing  the  KRAs  generated  in  group  sessions. 


3.3.2  Initial  Briefing 


Assuming  the  steps  <-ec0mmended  in  the  implementation  plan  have  been  carried  out,  participants 
should  be  aware  of  the  purposes  of  the  measurement  process  for  their  organization,  Next,  an 
initial  briefing  shoulo  be  provided  at  the  MCT  meetings.  This  briefing  should  accomplish  the 
following  objectives: 

1.  Explain  the  steps  in  the  MGEEM  measurement  development  process; 

2.  Create  the  proper  mental  set  for  KRA  development;  and 

3.  Explain  KRAs  and  discuss  their  role  in  the  MGEEM  process. 

A  recommended  agenda  and  example  of  this  initial  briefing  are  provided  in  Appendix  D. 


3.3.3  Posing  the  Question 


Tne  facilitator  uses  the  nominal  group  technique  (NGTj  (see  Delfcecq,  van  de  Ven,  S  Gustafson, 
1975)  to  initiate  the  MGEEM  process  by  writing  a  question  to  be  answered  on  a  sheet  of  chart 
paper  and  hanging  It  on  the  wal1  where  it  can  tc  seen  by  all  participants.  In  posing  this 
question,  the  facilitator  should  say: 

THIS  IS  THE  QUESTION  I  WOULD  LIKE  YOU  TO  ANSWER.  (At  this  point,  write  the 
following  question  on  the  chart  paper.)  "What  types  or  categories  of  results  does 
the  Air  Force  expect  this  organization  (name  the  target  organization)  to 
accompl 1 sh?" 

AS  YOU  THINK  ABOUT  THIS  QUESTION,  ALL  OF  YOU  SHOULO  PUT  YOURSELVES  IN  THE  POSITION 
OF  THE  COMMANDER  OF  YOUR  ORGANIZATION.  IN  OTHER  WORDS,  TODAY,  YOU  ARE  PLAYING  DOSS) 
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3.3.4  Silent  Generation  of  Answers 


Participants  are  tnen  asked  to  "silently  generate"  answers  to  the  question  posed  by  the 
facilitator.  This  process  is  important  since  the  resulting  answers  serve  as  the  primary  content 
for  the  remainder  of  the  process.  Sufficient  time  snould  be  allowed  sc  that  participants  do  not 
feel  rushed  and  have  time  to  reflect  on  their  answers.  Usually  10  to  IS  minutes  is  sufficient. 
To  begin  this  process,  the  facilitator  should  say: 


ON  A  SHEET  OF  PAPER  WRITE  A  LIST  OF  YOUR  ANSWERS  *0  THIS  QUESTION.  PLEASE  00  THIS 
INDIVIDUALLY  WIThOUT  DISCUSSION.  TAKE  AS  MICH  TIME  AS  YOU  NEED  TO  FELLY  ANSWER  THE 
QUESTION.  DOES  ANYONE  HAVE  ANY  PROBLEMS  WITH  THE  QUESTION  AS  IT  IS  STATED?  (At 
this  point,  sorae  participants  may  want  to  change  a  word  or  discuss  tne  question. 
Minor  changes  in  wording  can  be  made  in  order  to  help  participants  accept  the 
question.  It  is  important  that  parti cipants  become  committed  to  the  task  at  this 
point.)  IF  THERE  ARE  NO  FURTHER  QUESTIONS,  THEN  BEGIN. 


3.3.5  Round-Robin  listing 

In  this  activity,  answers  generated  by  individual  participants  in  the  first  phase  are 
recorded  on  chart  paper  as  quickly  as  possible.  This  process  is  conducted  without  comment  or 
discussion,  except  for  clarification  in  order  to  accurately  and  legibly  record  answers.  The 
facilitator  moves  around  the  group  in  round-robin  fashion,  getting  one  answer  from  each 
participant.  This  process  continues  until  all  participants  say  "pass."  indicating  that  all  of 
their  answers  have  been  recorded. 

To  begin  this  process,  the  facilitator  should  so/: 

NOW  WE  WCI'lO  like  TO  HAVE  rou  READ  YOUR  ANSWERS  ONE  AT  A  TIME  SO  THAT  WE  CAN  RECORD 
THEM  ON  CHART  PAPER,  THE  OBJECTIVE  IS  TO  WRITE  Y  OUR  ANSWERS  AS  QUICKLY  AS 
POSSIBLE;  THEREFORE,  WE  WILL  NOT  HAVE  ANY  DISCUSSION  QP  ANSWERS  AS  THEY  ARE 
WRITTEN,  EXCEPT  TO  CORRECT  INACCURACIES.  I  WILL  START  WlTh  -  (name  a  participant) 

-  THEN  PROCEED  AROUND  THE  TABLE  IN  R0UND-RC8 IN  FASHION  UNTIL  ALL  YOl'R  ANSWERS  HAVE 
BEEN  TRANSFERRED  TC  THE  CHART  PAPER.  (Name  a  participant) ,  WILL  YOU  OWE  ME  ONE 
ANSWER  OFF  YOUR  LIST?  (Once  the  answer  Is  written,  ask  the  participant  if  the  item 
is  Ok.  If  not,  then  make  any  necessary  corrections.  If  it  is  OK,  then  proceed  to 
the  next  person  and  write  his/her  first  answer.  Continue  in  this  manner  until  all 
answers  are  on  the  chart  paper. ) 

A  sample  listing  of  original  KRAs  from  a  food  service  organization  is  shown  in  Table  3. 

3.3.6  Discussion  and  Clarification 

Once  all  answers  (KRAs)  are  listed,  the  facilitator  leads  the  group  through  a  review 

process.  The  purpose  of  the  review  process  is  to  provide  an  opportunity  to  necify  KRAs  so  as  to 
improve  their  clarity  and  accuracy  and  --educe  overlap.  At  this  t  ,;e,  participants  may  ask 

questions  to  have  the  meaning  of  KRAs  explained.  Another  purpose  of  the  renew  process  is  to 

combine  KRAs,  as  appropriate,  to  remove  redundancy  or  to  achieve  a  consistent  level  of  KRA 

specificity.  KRAs  may  be  combined  if  they  are  too  narrow  or  specific.  Consistent  specificity 
can  be  achieved  by  breaking  KRAs  that  are  too  broad  into  two  or  more  r.RAs.  As  -  guide,  a  typical 
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Initial  KRA  list  might  contain  from  30  to  50  answers.  Following  discussion  and  clarification, 
this  number  should  be  reduced  by  20i  to  401.  The  modified  list  of  original  KRAs  in  Table  3  is 
shown  In  Table  4. 


Table  3.  Original  Key  Result  Areas  (KRAs)  -  Food  Service 


1. 

2. 

3. 

4. 

6. 

7. 

8. 
9. 

10. 

11. 

12. 

13. 

14. 


15. 

16. 

17. 

18. 

19. 

20. 

21. 

22. 

23. 

24. 

25. 

26. 

27. 

28, 

29. 

30. 

31. 

32. 


Provide  quality  meals  to  customers. 

Best  possible  variety  of  food. 

Ensure  effective  training  programs. 

Quality  facilities. 

Equipment  turnover  -  5-year  plan  for  replacement  of  equipment. 

Organize  and  manage  supervisors  to  operate  facilities. 

Maintain  acceptable/sanitary  housekeeping  conditions. 

Maintain  effective  relationships  with  other  base  agencies. 

Satisfy  customers  through  consistent  quality,  variety,  service,  ar.d 
quality  facilities. 

Maintain  adequate  manpower  to  reach  objectives. 

Effective  quality  assurance  evaluation. 

Control  and  safeguard  Inventory  -  food  and  supplies. 

Provide  sufficient  personnel  to  satisfy  deployment  needs  without 
sacrificing  production  at  home  or  exceeding  normal  work  hours. 

Set  a  good  example  for  people  to  follow  -  support  the  peop'e;  maintain 
standards;  reward  and  recognize  people;  maintain  morale,  esprit,  and  job 
satisfaction;  and  maintain  quality  facility  for  staff. 

Meet  the  required  schedules  and  respond  to  emergency  needs. 

Support  the  wing/base  mission,  and  maintain  effective  relationships 
with  other  agencies. 

Make  productive/efficient  use  of  available  resources;  obtain  adequate 
funding  for  equipment/supplies. 

Maintain  fiscal  accountability. 

Continue  to  Improve  operations  rather  than  stagnate. 

Compliance  with  regulations;  e.g.,  fiscal  accountability,  meeting 
suspenses,  inventory  control,  sanitation,  budgeting,  and  obtaining  funding. 
Satisfy  wing  commander. 

Be  versatile,  creative,  Innovative,  and  imaginative. 

Support  his/her  people. 

Food  service  long-range  planning. 

Maintain  required  documentation. 

Be  able  to  tell  your  boss  “like  it  is." 

Manage  the  gray  areas. 

Identify,  report,  and  follow  up  on  equipment  maintenance. 

Develop  realistic  budgets  and  obtain  funding. 

Provide  satisfactory  workplace  for  our  people. 

Provide  quality  food  at  c  rrect  selling  price. 

Zero  defects  on  fraud,  waste,  and  abuse. 
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Table  4.  Original  Key  Result  Areas  (KRAs)  -  Food  Service 
Following  Discussion  and  Clarification 


Q  1. 
0  2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 


10. 

11. 

12. 

13. 

14. 


15. 

16. 

17. 


18. 

19. 

20. 

21. 


Provide  quality  weals  tc  customers. 

Qest  possible  variety  of  feed. 

Ensure  effective  training  programs. 

Quality  facilities. 

Equipment  turnover  -  5-year  plan  for  replacement  of  equipment. 

Orgariee  and  manage  supervisors  to  operate  faeH-lticei 
Maintain  acceptable/sanitary  housekeeping  conditions. 

Maintain  effective  relationships  with  other  base  agencies. 

Satisfy  customers  through  consistent  quality,  variety,  service,  and 
quality  facilities. 

Maintain  adequate  manpower  to  n a eh  objectives. 

Effective  quality  assurance  evaluation. 

Control  and  safeguard  inventory  -  food  and  supplies. 

Provide  sufficient  personnel  to  satisfy  deployment  needs  without 
sacrificing  production  at  home  or  exceeding  normal  work  hours. 

Set  good  example  for  people  to  follow  -  support  the  people;  maintain 
standards;  reward  and  recognize  people;  maintain  morale,  esprit,  and  job 
satisfaction;  and  maintain  quality  facility  for  staff. 

Meet  required  schedules  and  respond  tc  emergency  needs. 

Support  the  wing/base  mission,  and  maintain  effective  relationships  with 
other  agencies. 

Make  productive/efficient  use  of  available  resources;  obtain  adequate 
funding  for  equipment/supplies. 

Maintain  fiscal  accountability. 

Continue  to  Improve  operations  rather  than  stagnate. 

Compliance  with  regulations;  e.g.,  fiscal  accountability,  meeting 
suspenses,  Inventory  control,  sanitation,  budgeting,  and  obtaining  funding 
Satisfy  wing  commander. 

Be  versatile,  creative,  innovative,  and  imaginative. 

Support  hia/hee  t»eoplo. 

Food  service  long-range-  planning< 

Maintain  required  documentation 
Be  aisle  to  tell  your  boss  d-sr* 

Manage  the  gray  areas. 

Ideatlfy-r-nopanti  and  follow  up  on  egvipwent  maintenance. 

Develop  recti atle  badge  tc  and  obtain  funding^ 
fvovldo  aatlafaetony  workplace  for  our  people. 

FVovIde  qwal  1ty--Food -at  correct  soiling  prlcei 
Zei*e  defects  an  fraud,  waste,  and  abocer 


3.3.7  Vote  1 


Participants  are  next  asked  to  select  KRAs  from  the  original  list  which  they  feel  are  the 
most  Important  for  the  unit.  The  facilitator  determines  the  number  to  be  selected.  Generally, 
havirg  the  group  select  5  to  7  is  reconmended.  Although  limiting  KRAs  to  this  number  is 
difficult.  It  helps  the  group  focus  on  an  appropriate  level  of  specificity.  It  also  serves  to 
highlight  areas  of  agreement  or  disagreement  within  the  group  with  respect  to  the  priorities  of 
the  organization. 

The  facilitator  can  conduct  the  voting  process  as  follows: 
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NOW  THAT  YOU  HAV  £  HAT  AN  OPPORTUNITY  TO  DISCUSS  THE  KRAs  ,  IT  IS  ' ! HE  TO  PRIORITIZE 
THEM.  PLEASE  BEAR  WITH  ME  BECAUSE  THIS  VOTING  PROCESS  IS  A  BIT  TEDIOUS.  HOWEVER, 
IT  IS  IMPORTANT  THAT  YOU  FOLLOW  THE  PROCESS  EXACT l  Y.  YO’J  WILL  ISE  ThE  INDEX  CARDS 
ON  THE  TABLE.  EACH  PERSON  ShOULO  HAVE  SEVEN  (7)  INDEX  CARDS. 

FIRST.  LOON  AT  THE  KRAs  WHICH  REMAIN.  (At  this  point,  the  facilitator  circles  in 
red  on  the  chart  paper  the  numbers  of  the  KRAs  eligible  for  voting),  FROM  THIS 
LIST  OF  KRAs,  SElECT  The  SEVEN  WHICH  YOU  FEEL  ARE  MOST  IMPORTANT  IN  TERMS  OF 
ANSWERING  THE  QUESTION  WE  INlTiALLY  POSED.  REMEMBER.  AS  YOU  MAKE  THIS  SELECTION, 
YOU  SHOULD  VIEW  THIS  PROCESS  FROM  THE  UNIT  COMMANDER’S  VIEWPOINT.  THAT  IS,  Wh.CH 
OF  THESE  ITEMS  WOULD  THE  COMMANDER  CONSIDER  MOST  IMPORTANT?  ONCE  YOU  have  SELECT  ED 
SEVEN  KRAs,  WRITE  EACH  KRA  AND  ITS  NUMBER,  ONE  AT  A  TIME,  ON  THE  INDEX  CARDS  YOU 
HAVE  BEEN  GIVEN.  WRITE  THE  KRA  NUMBER  IN  THE  UPPER  LEFT  CORNER  OF  THE  CARD  AND  THE 
KRA  ITSElF  It.  THE  MIDDLE  OF  THE  CARO.  (It  is  helpful  for  the  faciliator  to  draw  a 
sample  card  on  the  cnart  paper  to  illustrate  where  to  write  the  KRA  and  its  number.) 


KRA 


WHEN  YOU  HAVE  COMPLETED  THIS  STEP,  LAY  OUT  ALL  ?  -ARDS  ON  THE  TABLE  IN  rRONT  OF  YOU. 

CONSIDER  ONLY  THE  SEVEN  CARDS  IN  FRONT  OF  YOU.  OF  THESE  SEVEN  KRAs,  WHICH  IS  MOST 
IMPORTANT?  SELECT  THAT  KRA  AND  WRITE  THE  NUMBER  ’  IN  THE  LOWER  RIGHT  CORNER  OF  THE 
CARO.  TURN  THAT  CARO  OVER.  (The  facilitator  should  illustrate  this  by  writing  the 
number  in  the  lower  right  corner  of  the  card  drawn  on  the  chart  paper.) 


i 


KRA 


I 


OF  THE  SIX  REMAINING  KRAs.  WHICH  IS  LEAST  IMPORTANT?  WRITE  THE  NUMBER  1  IN  THE 
LOWER  RIGHT  CORNER  OF  THAT  CARO.  TURN  THE  CARD  OVER.  OF  THE  FIVE  REMAINING  KRAs, 
SELECT  THE  ONE  THAT  IS  MOST  IMPORTANT  AND  WRITE  A  6  IN  THE  LOWER  RIGHT  CORNER  OF 
THE  CARC.  TURN  THE  CARD  OVER.  OF  THE  FOUR  REMAINING  KRAs,  SELECT  THE  ONE  THAT  IS 
LEAST  IMPORTANT  AND  WRITE  A  2  IN  THE  LOWER  RIGHT  r0RNER.  TURN  THE  CARD  OVER.  OF 

the  remaining  three  kras,  select  the  one  that  is  most  important  anc  -..rite  a  5  in 

THE  LOWER  RIGHT  CORNER  OF  THE  CARD.  TURN  THE  CARD  OVER.  OF  THE  REMAINING  TWO 
tRAs,  SELECT  THE  ONE  THAT  IS  LEAST  IMPORTANT  AND  WRITE  A  2  IN  T uE  LOWER  RIGHT 
CORNER.  TURN  THE  CARD  OVER.  WRITE  A  4  IN  THE  LOWER  RIGHT  CORNER  OF  THE  REMAINING 
CARD.  THIS  COMPLETES  THE  VOTING  PROCESS.  PLEASE  GIVE  ME  YOUR  CARDS.  (As  the 
vote:  are  tallied,  it  is  a  convenient  time  to  give  the  group  a  break.  The  tally 
can  te  --ecorded  as  shown  In  Table  5.) 


IB 


Table  5.  Food  Service  Key  Result  Areas  -  Vote  1  Results 


(Voting 
feel  are 

Instructions  to  eight  group  members: 
most  important) 

Select  tne  seven  KRAs  that  , 

KRA  NO. 

VOTING  PATTERN 

voting  SUMMARY 

3. 

5-2 -4-4  -4 

5/19a 

4. 

• 

• 

m 

• 

4/9 

5. 

4-1 -5-1 -3 

5/14 

7. 

2-5 

2/7 

8. 

4 

1/4 

9. 

7-7-6-6-1-7-5-7 

8/46 

11. 

3-2 

2/5 

12. 

3 

1/3 

13. 

2-2 

2/4 

14. 

15. 

6-4-7 -6-2-6 

6/31 

16. 

6-S-7-7-7 

5/32 

17. 

3-S-2-6-6 

5/22 

18. 

3-1-3 

3/7 

19. 

3 

1/3 

20. 

21. 

2-4-1 

3/7 

22. 

1-6-5 

3/11 

indicates  that  five  of  the  eight  participants  selected  kra  Number  3. 
and  the  sum  of  Us  assigned  weights  was  19  (5+2+4+4»4 ). 


3.3.8  Tallying  the  voting  Results 


The  facilitator  tallies  the  results  of  the  voting  as  follows: 

1.  Generate  a  chart  paper  sheet,  listing  In  the  left  column  In  ascending  order  the  numbers 
cf  all  KRAs  that  survived  the  discussion  and  clarification  process.  An  example  is  shown  in  Tatle 
6  under  the  column  marked  "KRA  NO."  Note  in  Table  4  that  KRAs  1,2,6,  10,  and  23  through  32  did 
not  survive  the  discussion  and  clarification  process  and,  consequently,  do  not  appear  In  Table  5. 

2.  Sort  the  Index  cards  from  the  eight  participants  Into  stacks  according  to  the  KRA  numbers 
In  the  upper  left  corners  of  the  cards. 

3.  Pick  up  the  stack  of  Index  cards  with  the  lowest  r.RA  number.  In  Table  5,  the  lowest 
remaining  KRA  number  Is  3.  For  the  Index  cards  In  this  stack,  record  in  the  middle  column  on  the 
chart  paper  (the  column  in  Table  5  labeled  "VOTING  PATTERN")  the  weights  assigned  In  vote  1  by 
the  team  members.  These  weights  are  the  numbers  In  the  lower  right  corner  of  the  stack  of  cards 
for  KRA  3.  As  shown  In  Table  5,  KRA  3  was  selected  by  five  participants  who  gave  weights  of 
5 -2-4-4  -4. 

4.  Finally,  record  In  fractional  form,  In  the  right  column  of  the  chart  paper  (In  Table  5, 
this  Is  ’abeled  “VOTING  SUMMARY"),  a  summary  of  the  voting  results  for  the  KRAs.  The  numerator 
Is  the  number  of  team  members  who  selected  the  KRA  as  Important,  and  the  denominator  Is  the  sum 
of  the  weights  assigned  by  the  team  members  who  thought  the  KRA  was  Important.  The  fraction  for 
KRA  3  Is  5/19.  Five  team  members  Identified  KRA  3  as  important,  and  the  sum  of  their  assigned 
weights  Is  19  (5+2+4+4+41. 

5.  Continue  this  process  for  each  kRA.  An  example  of  results  Is  shown  In  Table  5. 


WSAi*  Oilers--: 
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3.3.9  Discussion  of  Vote  1 


The  discussion  that  follows  the  first  vote  Is  ar  Important  step  In  the  MGEEM  process  with 
respect  to  (a)  the  quality  of  the  resulting  product  and  (b)  the  degree  of  participant  commitment 
to  the  result.  If  there  is  complete  consensus  following  vote  1,  this  discussion  is  unnecessary. 
However,  complete  consensus  is  rare  at  this  point.  Therefore,  the  purpose  of  this  discussion  is 
to  bring  out  additional  information  that  will  promote  consensus  and  modify  KRAs  in  ways  that  will 
promote  greater  consensus.  The  facilitator's  role  Is  to  guide  the  group  through  this  discussion, 
always  focusing  on  the  KRAs  and  on  ways  to  make  them  more  meaningful,  less  ambiguous,  or  more 
accurately  stated.  The  facilitator  should  use  the  voting  pattern  from  vote  1  as  the  starting 
point  for  this  discussion.  The  discussion  should  center  around  KRAs  for  which  more  than  one 
participant  voted,  but  which  show  a  reasonably  wide  spread  In  the  voting  pattern.  For  example. 
In  discussing  the  voting  pattern  shown  in  Table  5,  the  facilitator  might  say: 

NOW  WE  ARE  GOING  TO  DISCUSS  THE  VOTING  PATTERN  THAT  RESULTED  FROM  YOUR  FIRST  VOTE. 

AS  YOU  CAN  SEE,  THE  GROUP  DOFS  NOT  SEE  THINGS  EXACTLY  Thfc  SAME  WAY.  IN  THIS 
DISCUSSION,  WE  WILL  BE  ATTEMPTING  TO  MAKE  THE  KRAs  BETTER,  NOT  "TWIST  YOUR  ARMS"  TO 
GET  YOU  TO  VOTE  IN  A  CERTAIN  WAY.  HOWEVER,  IT  IS  OUR  GOAL  TO  ACHIEVE  AS  MUCH 
AGREEMENT  AS  POSSIBLE  AMONG  GROUP  MEMBERS. 

LET'S  LOOK  FIRST  AT  KRA  NUMBER  3.  FIVE  PEOPLE  VOTED  FOR  THIS  KRA  AND  THREE  PEOPLE 
010  NOT.  THOSE  WHO  SELECTED  IT  RATED  IT  S-2-4-4-4.  WOULD  SOMEONE  WHO  BELIEVES 
THAT  THIS  KRA  IS  IMPORTANT  PLEASE  TELL  US  WHAT  CAUSED  YOU  TO  SELECT  IT?  (Now  allow 
time  for  this  person  to  point  out  the  strengths  -  1  or  2  minutes  is  sufficient  - 
don't  allow  long  speeches.)  WOULD  ANYONE  ELSE  LIKE  TO  SAY  WHY  THEY  VOTED  FOR  THIS 
KRA?  (Continue  until  anyone  who  wants  to  talk  has  had  the  opportunity  to  argue 
brief  ly  for  the  KRA. ) 

NOW,  COULD  SOMEONE  WHO  010  NOT  SELECT  THIS  KRA  TELL  WHY  YOU  DIO  NOT  BElICVE  THIS  IS 
one  OF  THE  TOP  KRAs?  (Continue  the  discussion,  as  before,  until  all  issues  are 
aired.) 

NOW  THAT  WE  HAVE  HEARO  THIS  DISCUSSION,  WHAT  CHANGES  SHOULD  WE  MAKE  IN  THE  KRA? 
SHOULD  IT  BE  COMBINED  WITH  ANOTHER  KRA?  (The  facilitator  is  guided  by  the  group 
either  to  make  no  charges  to  the  KRA,  to  modify  it,  to  delete  It,  or  to  combine  It 
with  another  KRA.  If  a  KRA  Is  to  be  combined  with  another,  draw  a  line  through  the 
KRA  and  beside  It  write  the  number  of  the  KRA  with  which  it  is  being  combined, 
encircling  the  number  as  shown  In  Table  4.) 

The  discussion  should  seek  consensus  by  making  sure  group  members  share  a  common  under¬ 
standing  of  the  meaning  of  KRAs.  This  Is  done  by  allowing  participants  to  attempt  to  Influence 
each  other's  Judgments  by  adding  new  Information  about  KRAs,  such  as  history  of  the  organization 
or  future  trends  that  may  Influence  priorities,  customer  or  higher  headquarters  views,  etc.  This 
should  be  an  open  Information-sharing  and  clarification  session.  It  must  not  be  a  coercive, 
arm-twisting  discussion. 


3.3.10  Vote  2 


Following  the  discussion,  the  voting  process  carried  out  in  3.3.7  is  repeated  for  the 
remaining  KRAs.  Again,  five  to  seven  KRAs  should  be  selected  as  the  most  important.  If  the 
facilitator  believes  that  there  are  Important  unresolvable  di sagreements  among  the  participants, 
the  number  of  KRAs  may  be  Increased  from  7  to  9,  although  this  is  us-.ally  unnecessary.  The 
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number  of  KRAs  voted  on  1$  not  necessarily  the  number  of  KRAs  that  will  eventually  be  retained 
and  recommended  to  management.  In  this  example,  the  KRAs  eligible  for  vote  2  would  be  all  those 
listed  In  Table  5  except  Nos.  15  and  21,  which  were  not  selected  by  any  participant  In  vote  1. 
Those  selected  by  vote  2  are  shown  In  Table  6. 


Tab! e  6,  Food  Service  Key  Result  Areas  *  Vote  2  Results 


KRA  NO. 

VOTING  PATTERN 

VOTING  SUMMARY 

RANK 

3 

2-4-1 -6-3-2-3-3 

8/24 

5.5 

5 

I-5-6-3-2-3-2-2 

8/24 

5.5 

9 

6-7-7-7-4-6-6-7 

8/50 

1 

1 3a 

3-4-4-1 

4/12 

8 

14 

4-6-3-7-S-5-6 

7/36 

2 

16 

3-1. 5.5.I  .7.7.5 

8/34 

3 

17 

7-1-6-4-4-4 

6/26 

4 

20 

5-2-2— 2-5-1 -1 

7/18 

7 

4KRA  dropped  by  consensus  following  vote  2. 


3.3.11  Documenting  Results 

Following  the  second  vote,  sufficient  consensus  Is  normally  achieved,  and  the  process  can  be 
ended.  If  not,  the  facilitator  repeats  steps  3.3.9  and  3.3.10  until  sufficient  agreement  Is 
obtained.  Once  the  process  Is  terminated,  the  facilitator  documents  the  process.  Forms  for 
recording  the  KRA  development  process  are  provided  In  Appendix  E.  A  format  for  reporting  the 
final  results  of  the  session  Is  shown  In  Table  7. 


Table  7.  Food  Service  -  Final  Key  Result  Areas 


VOTE 

RANK 

KRA 

(max  *  8/56) 

8/50 

1 

Customer  Satisfaction.  Satisfy  customers 
through  consistent  quality  and  variety  of 
food  and  service  and  through  providing  and 
maintaining  quality  facilities. 

7/36 

2 

Human  Resource  Management/Leadership.  Maintain 
a  motivated  work  force  with  high  morale,  as* 
prlt,  and  Job  satl sfactlon;  Involve  work  force 
In  decision  making;  Implement  creative  and 
Innovative  practices;  encourage  high  perfor¬ 
mance  through  recognizing  and  rewarding 
personnel;  and  provide  leadership  by  example. 

8/34 

3 

Support,  Maintain  productive  relationships 
with  other  base  agencies  supported  and 
upon  whom  the  organization  depends  for 
support. 

6/26 

4 

Productivity  of  Resources.  Make  productive/ 
efficient  use  of  all  available  resources. 

8/24 

5.5 

Equipment  Availability.  Provide  equipment  re- 
qulred  to  accomplish  the  food  service  mission. 

8/24 

S.5 

Training  Effectiveness.  Ensure  that 
personnel  are  effectively  trained. 

7/18 

7 

Administrative  Compliance.  Comply  with  Air 
Force/MAJCOM  regulations  and  requirements. 
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3.4  Developing  Indicators 


The  next  step  in  the  MGEEM  process  is  the  development  of  performance  indicators.  Performance 
indicators  are  the  measurement  tools  used  to  know  if  KRAs  are  being  accomplished.  Indicators  may 
take  several  forms,  although  they  typically  are  ratios  which  compare  some  aspect  of  performance 
(e.g.,  number  of  transactions  processed  successfully)  to  some  reference  point  (e.g.,  total  number 
of  transactions  processed]  or  to  some  measure  of  resource  input  (e.g.,  number  of  hours  worked). 
In  practice,  It  is  common  to  relate  the  ratio  for  a  point  in  time  to  a  base  period  to  detect 
changes  In  an  indicator  from  one  measurement  period  to  another,  in  the  MGEEM  process,  Indicator 
development  is  a  multi-step  process.  The  steps  involve  Indicator  definition,  prioritization  of 
Indicators,  management  review  and  refinement,  and  implementation.  These  steps  are  discussed  in 
the  sections  which  follow. 


3.4.1  Convening  the  Indicator  Development  Team 


The  Indicator  development  team  Is  Team  B.  Considerations  in  selecting  the  members  of  this 
team  are  outlined  in  Section  3.2.  Once  selected,  the  indicator  development  team  members  are 
invited  to  the  indicator  development  session.  This  invitation  should  come  from  the  unit 
commander  of  tne  target  organization.  The  invitation  memo  or  letter  should  contain  Information 
about  the  time  and  location  of  the  session,  its  purpose,  and  the  amount  of  time  required. 


3.4.2  Briefing  Team  B  Members 

Team  B  members  should  be  given  the  overview  briefing  given  to  Team  A  (see  Section  3.3.2).  In 
addition,  the  Initial  briefing  of  Team  B  should  Include  Identification  of  the  characterl sties  of 
good  Indicators,  an  overview  of  Indicator  types,  and  sample  Indicators  from  other  Air  Force 
organizations.  A  recommended  agenda  and  briefing  are  in  Appendix  f.  Following  the  briefing, 
Team  B  members  should  be  given  the  quiz  In  Appendix  G  to  serve  as  a  review  and  to  heighten  tne 
distinction  between  efficiency  and  effectiveness.  The  facilitator  should  stress  this  distinction 
because  Air  Force  organizations  tend  to  pay  more  attention  to  effectiveness  (accomplishing 
mission  objectives)  than  to  efficiency  (the  best  use  of  resources  to  produce  products  and 
services). 


3.4.3  Olscusslon  of  KRAs  Developed  by  Team  A 


Prior  to  indicator  development.  Team  B  members  should  review  the  final  KRAs  developed  Dy  Team 
A.  In  the  example  of  KRA  development  given  above,  Team  B  would  be  asked  to  review  the  data 
preserted  in  Table  7.  The  facilitator  should  ask  one  of  the  Team  8  members  who  was  also  a  Team  A 
member  to  comment  on  each  of  the  KRAs,  The  purpose  of  this  review  is  not  .o  modify  KRAs,  but  to 
provide  Team  B  members  with  a  framework  for  thinking  about  the  mission  of  the  organization  and  to 
achieve  a  sense  of  their  "ownership"  of  the  KRAs.  If  the  KRAs  are  presented  as  a  fait  accompli 
to  Team  B,  some  resistance  may  develop. 


3.4.4  Using  the  Nominal  6roup  Technique  (NGT)  to  Develop  Indicators 

The  NGT  process  is  again  used  to  develop  Indicators,  The  process  begins,  as  with  KRA 

development,  with  the  facilitator’s  writing  on  chart  paper  the  question  tc  be  answered  and 

nanglng  the  sheet  on  the  wall  where  all  can  sen  It,  The  facilitator  begins  by  sa/irig; 
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OUR  TASK  IS  TO  DEVELOP  A  SET  OK  INDICATORS  THAT  WILL  ALLOW  US  TO  DETERMINE  IF  THE 
ORGANIZATION  IS  ACHIEVING  THE  RESULTS  DESCRIBED  IN  THE  LIST  OF  KRAs.  WE  WILL 
ACCOWyjSH  THIS  BY  HAVING  THIS  GROUP  ANSWER  THE  FOLLOWING  QUESTION  FOR  EACH  KRA. 

THE  QUESTION  IS;  (At  this  point  the  facilitator  writes  the  question  on  chart  paper 
or  reveals  the  question  that  has  previously  been  written.)  what  quantitative 
Indicators  should  the  commander  tract  on  a  periodic  basis  to  tell  If  this  KRA  Is 
being  accomplished?  WE  WILL  ANSWER  THIS  QUESTION  FOR  EACH  OF  The  KRAs. 

Once  members  of  the  group  have  indicated  their  understanding  and  acceptance  of  this  question, 
the  facilitator  begins  the  process  by  selecting  the  KRA  that  will  be  easiest  to  measure. 
Experience  shows  that  there  Is  a  learning  curve  with  the  MGEEM  Indicator  development  process. 
The  first  time  through  the  steps,  Team  B  members  are  usually  unsure  of  what  is  expected  of  them. 
The  KRA  for  which  indicators  are  easiest  to  develop  will  be  one  that  Is  concrete,  product-  or 
service-oriented,  and  one  for  which  there  are  currently  existing  indicators.  In  most  Air  Force 
organizations  there  are  reporting  requirements  to  higher  headquarters  on  principal  workloads. 
Selecting  a  kRA  that  pertains  to  work  products  currently  tracked  Is  advised.  The  facilitator  may 
wish  to  allow  Team  B  to  make  the  decision  about  which  kRA  to  use  to  start  the  Indicator 
development  process. 

For  example,  the  facilitator  could  say;  WE  WOULD  LIKE  TO  BEGIN  WITH  *h£  kRA  THAT 
WILL  BE  EASIEST  TO  MEASURE.  THIS  WILL  HELP  US  BECOME  MORE  COMFORTABLE  WITH  THE 
PROCESS.  WHICH  KRA  DO  YOU  SUGGEST? 

3.4.4. 1  Silent  Generation  of  Answers.  Team  members  are  next  asked  to  use  the  indicator 
development  worksheet  In  Appendix  H  as  they  think  of  indicators  for  the  selected  KRA.  They 
should  be  Instructed  to  think  of  both  efficiency  and  effectiveness  Indicators  and  write  them  on 
the  worksheet.  The  facilitator  should  allow  sufficient  time  for  participants  to  consider  and 
list  Indicators  without  discussion  among  themselves.  Team  members  should  be  reminded  that  this 
Is  an  Individual,  silent  task.  The  opportunity  for  discussion  will  come  later.  To  begin  the 
process  the  facilitator  could  say: 

USING  THE  INDICATOR  OEVEcOPMENT  WORKSHEET,  PLEASE  l  1ST  THE  EFFICIENCY  AND 
EFFECTIVENESS  INDICATORS  THAT  YOU  FEEL  ARE  ANSWERS  TO  THE  QUESTION  WRITTEN  ON  THE 
CHART  PA^ER.  PLEASE  DO  THIS  INDIVIDUALLY  WITHOUT  DISCUSSION.  TAKE  AS  MUCH  TIME  AS 
YOU  NEED  TO  FULLY  ANSWER  THE  QUESTION.  BE  SURE  TO  TRY  TO  DEVELOP  BOTH  EFFICIENCY 
AND  EFFECTIVENESS  INDICATORS.  ARE  THERF  ANY  QUESTIONS?  (Take  time  to  answer  any 
questions  posed  by  participants.)  IF  NOT,  THEN  BEGIN, 

3. 4. 4. Z  Round-Robin  Listing.  The  next  step  is  to  transfer  the  Indicators  that  members  have 
generated  onto  chart  paper  so  that  all  members  of  the  team  can  review  and  discuss  them.  The 
facilitator's  task  here  is  to  move  around  the  group,  getting  one  Indicator  per  person,  and  on  the 
chart  paper,  writing  that  Indicator  exactly  as  stated.  After  writing  tne  Indicator,  the 
facilitator  should  confirm  that  the  indicator  Is  properly  recorded  by  asking  the  team  member  who 
offered  It  If  the  Indicator  Is  written  correctly.  Once  the  Indicator  Is  approved,  then  the 
facilitator  moves  to  the  next  member,  and  so  on  until  all  memDers  say  "Pass."  The  result  of 
round-robin  listing  mlgnt  appear  as  shown  In  Table  8. 
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Table  8.  Original  Indicators  -  Supply 


KRA  No.  1  -  Customer  Satisfaction 

1.  Average  Response  Time/Expected  Response  Time 

2.  Customer  Wait  Time  for  Issues 

3.  No.  of  Customer  Inquiries/No.  of  Productive  Hours  in  Customer 

Assistance  Function 

4.  Fill  Rate  on  Requisitions 

5.  no.  of  Documented  Customer  Complaints 

6.  Average  Turnaround  Time  from  Cate  of  Request  to  Date  of  Delivery 

by  Priority  Category 

7.  No.  of  Complaints  About  Quality  of  the  Product  That  Are 

Resolved/Total  No.  of  Complaints  About  Product  Qualitj 

8.  Percentage  of  Customers  Responding  to  Survey  who  Report  Satisfaction 

with  Supply  Service 

9.  No.  of  Complaints  Received/No.  of  Transactions  Processeo 

10.  Average  Requisition  Processing  Time 

11.  Customer  Inquiries  Sati sf ied/CuStomer  Inquiries  Received 

12.  Supply  Transactions  Processed  No.  of  Productive  Hours 


3. 4. 4. 3  Discussion  and  Clarification.  The  next  step  is  reviewing  the  list  of  indicators 
produced.  Here,  the  facilitator  leads  the  group  through  a  consideration  of  eaci  indicator  one  at 
a  time.  This  review  could  begin  with  the  facilitator's  saying: 

WE  NOW  HAVE  A  CHANCE  TC  DISCUSS  EACH  OF  THE  PROPOSED  INDICATORS.  AT  T'MS  STAGE  IN 
THE  PROCESS,  WE  ARE  TRYING  TO  REFINE  THE  INDICATORS  TO  MAKE  THEM  MORE  PRECISE.  W£ 

ARE  ALSO  CONCERNED  WITH  REDUCING  REDUNDANCY  AMONG  THE  INDICATORS.  NOW,  LET'S  LOOK 
AT  INDICATOR  NUMBER  1.  ARE  THERE  AN“  QUESTIONS  ABOUT  WHAT  THIS  INDICATOR  MEANS? 

HOW  OFTEN  WOULD  THIS  INDICATOR  BE  TRACKED  ?  WEEKLY?  MONTHLY?  QUARTERLY? 
ANNUALLY?  DOES  IT  PVERl AP  ANY  OTHER  INDICATORS  ON  THE  LIST?  IF  SC,  WHICH  ONE 
SHOULD  WE  KEEP? 

The  facilitator  should  proceed  in  this  manner  through  the  list  of  indicators.  At  the 
beginning  of  the  discussion  and  clarification  step,  there  will  be  a  list  such  as  that  shown  ‘n 
Table  8;  at  the  end  of  the  step,  the  list  might  look  like  that  in  Table  9. 
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Table  9.  Original  Indicators  -  Supply  -  Following  Olscusslon 
and  Clarification 


FRA  No.  1.  Customer  Satisfaction  Time  t>eriod 

1 .  Average  ftospwvoe  Response  Tiwc 

2.  Customer  Wait  Time  for  Issues  Monthly 

3.  No.  of  Customer  Inquiries/No.  of  Productive  hours  in  Customer 

Assistance  Function  Monthly 

4.  Fill  Rate  on  Requisitions  Monthly 

5.  N»r- »f -Documented  Cm Fowee- Complaints 

6.  Average  Tu-na round  ■ffme—Frem-Oat'e  ef- Request  te  Oate— »f 
Delivery  by  PrterFty  Categery 

7.  No.  of  ■Cowel»Fnts-At»ev4  ^ual4  t»  -ef-  the- Predict  That  Are 

41eseWve<w  lotr»4  -No.  of  -6e»p*+*a*fr-Ab«rt  ^redact  Qua-H-ty- 


8.  Percentage  of  Customers  Responding  to  Survey  Who  Report  Semi -A.^ually 

Satisfaction  with  Supply  Service 

9.  no.  of  Complaints  Received/No.  of  Transactions  Processed  Monthly- 

10.  Average  Requisition  p-ocessing  Time  ■•’enthly 

11.  Customer  Inquiries  Satisfied/Customer  Inquiries  Received  Monthly 

12.  Supply  Transactions  Processed  No.  of  reductive  Hours  Monthly 


3. 4. 4. a  vote  1 .  This  step  of  the  M.GEEM  process  serves  to  eliminate  undesirable  and 
Impractical  indicators.  It  also  produces  a  ranking  of  indicators  with  Aspect  to  their 
importance  as  judged  by  Team  B  members.  This  Information  is  valualle  tc  the  comnander  as  a r 
indication  of  the  performance  areas  perceived  as  important  by  his  her  subordinates.  Tc  the 
extent  that  the  most  important  areas  arc  net  currently  being  tracked  by  f.e  organization, 
attention  should  be  dedicated  to  doing  so.  If  this  is  net  feasible,  then  the  commander  should 
explain  to  the  organization  why  it  is  not  necessary. 

If,  after  the  preceding  step,  there  are  less  than  five  Indicators,  their  r^.oeing  can  usually 
be  done  by  consensus.  Consensus  ranking  has  two  steps,  which  may  be  explained  : >  the  facilitator 
as  follows: 

OF  THE  REMAINING  INDICATORS,  WHICH  IS  THE  MOST  IMPORTANT  IE  RMS  OF  'USHERING  Tilt 
QUESTION  WE  INITIALLY  POSED?  (At  this  point  someone  will  say  a  -.mb c-r;  t.g., 
number  2.)  C.K.,  IS  THERE  ANY  OBJECTION  TO  LISTING  NUMEEK  2  AS  ThE  Mu  ST  IMPORTANT 

INDICATOR?  (If  no  objection  is  stated,  then  move  to  the  next  item  u  r  1 1 T  all  have 
been  ranked. ) 

Once  the  Indicators  have  been  ranked,  the  facilitator  asks  the  group  to  determine  how  many  „f 
the  ranked  indicators  are  of  sufficient  value  that  managimert  should  te  encou- :ged  to  track  them 
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on  an  ongoing  basis.  That  is,  the  group  should  decide  whether  the  organization's  resources 
should  be  committed  to  measuring  the  indicators.  This  feasibility  review  proceeds  in  reverse 
order  of  the  rankings.  Assuming  that  four  indicator;  had  been  ranked  by  consensus,  the 
facilitator  would  point  to  indicator  rank  number  tour  and  say: 

should  the  organization  commit  resources  to  capture  and  report  this  information? 
SHOULD  KE  keep  THIS  INDICATOR  ON  OUR  LIST? 

If  the  consensus  is  "no,"  then  drop  that  indicator  and  move  up  to  the  next.  The  process 
continues  until  an  indicator  is  encountered  that  at  least  some  members  feel  should  be  retained. 
Total  consensus  is  required  to  drop  an  indicator.  Since  the  Indicators  are  addressed  in  reverse 
order  of  importance,  the  first  indicator  that  the  group  decides  to  keep  ends  the  review  process, 
and  all  remaining  Indicators  are  considered  valid.  If  the  discussion  and  clarification  step 
results  in  more  than  five  indicators,  a  vote  is  recommended.  Prior  to  the  vote,  each  team  member 
should  be  given  a  set  of  five  index  cards.  The  facilitator  then  instructs  the  group  as  follows: 

please  review  the  indicators  that  remain  after  completion  of  the  discussion  and 

CLARIFICATION  STEP.  SELECT  THE  FIVE  INDICATORS  THAT  YOU  FEEL  AS  I.  THE  MOST 
IMPORTANT  IN  TERMS  OF  ANSWERING  THE  QUESTION  POSED  AT  THE  BEGINNING  OF  "'IS  PROCESS. 

ONCE  YOU  HAVE  SELECTED  FIVE  INDICATORS,  WRITE  THEM  -  ONE  PER  CARD  -  ON  THE  INDEX 
CARDS  PROVIDED.  THE  INDICATOR  STATEMENT  SHOULD  BE  WRITTEN  IN  Ti'E  CENTER  OF  THE 
CARD  AND  THE  NUMBER  OF  THE  I  NO  I  CAT  OR  FROM  ThE  CHART  PArER  SHOULD  EE  WT; !  'TEN  IN  THE 
UPPER  LEFT  CORNER  OF  THE  CARO.  ( -»t  this  point,  the  facilitator  should  illustrate 
this  on  chart  paper  as  shown  below.) 


No.  of  Complaints  Received 'Ho. 
of  Transactions  Processed 


NOW  THAT  YOU  HAVE  SELECTED  FIVE  INDICATORS,  SPREAD  ThE  FIVE  INDEX  CARES  OUT  CN  THE 
TABLE  IN  FROM  Ur  YUU  SO  ThA’  YOU  CAN  St  i  AIL  F I  VC  CARDS, 

COOKING  AT  ThE  FIVE  CARDS  IN  FRONT  OF  YOU,  TICK  THE  INDICATOR  THAT  IS  MOST 
IMPORTANT  IN  TERMS  OF  ITS  ANSWER  TO  THE  QUESTION  POSED  AT  THE  S’ART  OF  THIS 
SESSION.  ONCE  YOU  HAVE  SELECTED  THE  INDICATOR,  INDICATE  YOUR  CHOICE  BY  WRITING  A  5 
IN  THE  LOWER  RIGHT  CORNER  OF  THE  CARO,  YOUR  CARD  SHOULD  LOOK  LIKE  THIS.  (Using 
the  card  previously  drawn  on  chart  papar  (see  above),  write  a  i  in  the  lower  right 
corner  of  the  card  as  shown  below.) 


No.  of  Complaints  Received/No. 
of  transactions  Processed 
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NOW,  TURN  That  CARD  OVER.  FROM  THE  FOUR  REMAINING  indicators,  select  the  indicator 
that  is  least  important  IN  TERMS  OF  ns  ANSWER  TO  The  QLESTION.  INDICATE  YOUR 
CHOICE  BY  WRITING  A  1  IN  THE  LOWfR  RIGHT  CORNER  OE  THAT  i ARC.  TURN  THAT  LARD  OVER. 

NEXT,  FROM  THE  THREE  REMAINING  CAROS,  SELECT  THE  INDICATOR  THAT  IS  MOST  IMPORTANT 
AND  WRITE  A  4  IN  ThE  LOWER  RIGHT  CORNER.  TURN  THAT  CARD  OVER. 

FROM  THE  TWO  REMAINING  INDICATORS,  SELECT  THE  ONE  THAT  IS  LEAST  IMPORTANT  AND  WRITE 
A  2  IN  ThE  LOWER  RIGHT  HAND-  CORNER.  FOR  THE  REMAINING  INDICATOR,  WRITE  A  3  IN  THE 
LOWFR  RIC-HT  CORNER. 

After  this  vote  is  completed,  the  facilitator  collects  the  cards  and,  if  appropriate,  allows 
participants  to  take  a  short  break  while  the  vote  results  are  tallied.  If  a  break  is  not  needed, 
the  facilitator  directs  participants  to  begin  silent  generation  of  indicators  for  the  next  KRA. 
The  facilitator  should  tally  the  results  of  the  voting  according  to  the  procedure  described  i  n 
3.3.8:  (a)  Generate  a  chart  paper  sheet  with  the  numbers  for  ail  indicators  that  survived  the 

discussion  and  clari  fication  process  in  the  left  column  in  ascending  order.  Example  results  for 
an  8-member  team  arc-  shown  in  Table  10.  Note  under  the  columr  marked  "INDICATOR  NO”  that 
indicators  1,  E,  t,  ard  7  shown  in  Table  9  did  not  survive  the  discussion  and  clarification 
process  and,  consequently,  do  not  appear  i r.  Table  10.  (b)  Sort  the  index  cards  intc  stacks 

according  to  the  KRA  numbers  in  the  upper  left  corners  of  the  cards.  (c)  Pick  up  the  stack  of 
index  cards  with  the  lowest  indicator  number.  In  the  case  of  the  indicators  in  Table  10,  the 
lowest  indicator  number  is  2.  For  the  index  cards  in  this  stack,  write  in  the  second  column  cn 
the  chart  paper  (the  column  in  Table  10  mar«.ed  “VOTING  PATTERN" 3  tne  weights  assigned  in  vote  1 
by  the  team  members.  Tnese  weights  are  the  numbers  in  the  lower  right  corner  of  the  stack  ot 
cards  for  indicator  i.  Indicator  2  was  selected  by  four  of  the  sever,  participants,  who  gave  it 
weights  of  1-1-5-2.  Id)  Finally,  in  the  right  column  of  the  chart  paper  (see  Table  10)  labeled 
"VOTING  SUMMARY,"  write  the  fractions  which  summarise  the  voting  results  for  the  indicators.  Trc- 
numerator  is  the  number  of  team  members  who  selectee  the  indicator  as  important,  and  t'e 
denominator  is  the  sum  of  the  weights  assigned  by  the  team  members  who  thought  the  indicator  wus 
Important.  The  fraction  for  indicator  2  is  4/9.  hour  team  members  identified  indicator  2  as 
important,  and  the  sum  of  their  assigned  weights  is  9  (l+l»5+2). 

Table  10.  voting  Tally  Sheet  -  Indicators  Vote  1 


Function:  Sjpply 

xRA  No.  1.  Customer  Satisfaction 


indicator  no. 
2 

3 


voting  ■atiern 
l  -1  -5-2  ^ 

5-2-1  -4-1  -4-1  -2 


J 

a 

9 


1 1 
12 


3-3 

2- 3-2-! -1 -3 

3- 3-3-E-4 
5-5-2 

4- 4-5- 3-4-1 -4 

4-2-2-E 


VuUVj  Sl'ttth.-T 
4/9  ~~ 

8/2C 

2/C 

6/12 

5,  If. 


Royally  one  vote  is  sufficie-t  tc  screen  anc  prioritize  indicators.  However,  if  the 
facilitator  feels  trat  tie  group  has  net  adequately  discussed  the  indicators  or  if  there  is  a 
strong  split  in  tne  g'oup  (as  shown  by  the  voting  pattern),  then  additional  discussion  arj  a 
second  vote  i.  wa’-.'arted.  However,  the  neeo  for  d  second  vote  is  ra'e. 
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Following  the  tally,  the  facilitator  begins  another  cycle  by  picking  the  nc>.t  KRA  ana  having 
the  group  develop  individual  answers  to  the  question  of  how  best  to  measure  the  KkA  (see  3. 4. 4.1). 


3.4.5  Documenting  the  Process 


The  facilitator  must  ensure  tnat  the  total  process  is  documented.  The  indicator  development 
worksheet  in  Appendix  h  suggests  a  format  for  this  dccumentati on.  For  each  kRA,  the  facilitator 
should  record  the  original  indicators  suggested,  the  voting  pattern,  and  the  final  indicators. 
Table  11  is  an  example  of  a  final  indicator  report. 

Table  11.  Final  Indicators 


Key  Result  Area  No.  1  -  Customer  Satisfaction 


VOTING  SUMMARY 

RANK 

INC  1  l.ATCR 

(Max  *  8/40) 

1,21 

1 

No.  if  cost  icr  ir.q„ir;  ts  satisfied 
So.  of  cost-  mc-r  in  uirics  received 

■monthly ) 

8/30 

2 

so.  of  customer  inqulries/No.  cf 
P'uductive  itu'S  in  customer 
assistance  function  (monthly) 

5/18 

3 

No.  of  complaints  recei  veef/No.  of 
transactions  processed  (monthly) 

6/12 

4 

Percentage  of  customers  respond¬ 
ing  to  survey  who  report  satisfac¬ 
tion  with  supply  service  (semi¬ 
annual  ly ) 

3.5  Screening  Indicators  and  Identifying  Data  Sources 


Once  Team  6  has  completed  its  cask  of  defining  and  prioritizing  indicators  for  each  <RA,  tie 
indicators  are  reviewed  ty  the  management  of  the  target  organization.  The  essential  participants 
in  this  review  are  the  commander  c*  the  target  organization  and  the  measurement  facilitator.  Tri- 
commander  may  want  tc  include  in  this  review  group  the  person  who  is  most  knowledgeable  of  the 
administrative  practices  of  the  organization  and  otf.tr  >.iy  managers  wrose  support  is  needed  to 
implement  the  measurement  process. 


3.5.1  Preparing  Handout  Materials 


Prior  To  the  review  session,  the  facilitator  ensures  that  there  ar i  sufficient  copies  of  the 
final  indicator  ’ists  for  each  kRA  (Table  11)  to  pass  out  to  attendees.  These  lists  are  tr-e 
primary  input  to  this  review  session.  In  addition,  the  facilitaUr  should  ^ave  an  adequate 
number  of  indicator  review  sheets  'Appendix  I)  to  record,  the  results  bf  the  revitw  session.  C -e 
copy  of  the  indicate;  review  sheet  is  needed  for  each  participant,  and  tre  facilitator  needs  ore 
copy  for  each  indicator  so  Uat  nt  she  can  --ecord  results.  Ideally,  the  lists  of  indicators 
(Table  11)  can  be  distributed  to  -ill  participants  refore  the  meeting  sc  they  can  review  t'e 
indicators  in  advance. 


2  k 


3.5.2  Convening  the  Review  Session 


The  facilitator  should  schedule  a  review  session  with  the  unit  commander.  From  i  to  3  hours 
should  be  set  aside  for  this  review. 


3.5.3  Conducting  the  Review 


The  purpose  of  the  session  is  to  review  all  indicators  recommended  by  Team  S,  in  order  to 
select  those  to  be  used  to  measure  performance  ir.  the  target  organization.  In  reviewing 
indicators,  the  management  team  has  several  options:  delete  an  indicator,  modify  an  indicator, 
retain  an  indicator  as  stated,  or  defer  action  on  selection  until  further  research  is  done  to 
exandne  its  feasibility.  The  facilitator  should  press  for  decisions  on  all  indicators.  Part  of 
the  decision  process  relative  to  deleting,  retaining,  or  modifying  indicators  involves 
considering  sources  of  data  used  for  their  creation.  Therefore,  the  facilitator  reeds  to  capture 
the  data  source  as  specifically  as  possible  for  any  indicator  to  be  regained.  The  indicator 
review  form  establishes  structure  in  the  indicator  review  meeting.  To  begin  the  meeting,  the 
facilitator  distributes  i  review  form  to  each  participant,  then  says: 

OUR  PURPOSE  IS  TO  DEC  ICE  WHICH  INDICATORS  WIlU  UL  INCLUDED  IN  THE  FINAL  INDICATOR 
SET  FOR  THIS  ORGANIZATION  (The  flow  chart  of  tne  five  questions  shown  below  is 
presented  to  the  group  on  chart  paper.)  THIS  FLOW  CHART  HIGHLIGHTS  THE  IMPORTANT 
CONSIDERATIONS  IN  ObR  EVALUATION  OF  INDICATES.  FOR  EACH  INDICATOR,  I  WILL  ASK  YOU 
THE  FOLLOWING  ;lESTICNS: 

1.  CO  YCU  CURRENTLY  TRACK  THIS  INDICATOR  IN  THIS  FORM? 

2.  CO  YOU  CONSIDER  THIS  INDICATOR  IMPORTANT  10  TRAD*  ,  NICE  TO  HAVE,  CP  '.'NECESSARY 
TO  TRACK  OK  A  FORMAL  BASIS? 

3.  IF  THE  INDICATOR  IS  IMPORTANT  TC  TRACK,  Awe  YOU  SATISFIED  WITH  ThS  «AY  IT  IS 
WORDED? 

4.  DO  DATA  CURRENTLY  EXIST  TO  CREATE  THIS  INDICATOR?  IF  SO,  WHERE?  IF  i.CT,  HOW 
WILL  THE  DATA  dt  CAPTUREO? 

5.  IS  THE  INDICATOR  FEASIBLE  TO  TRACK?  IN  OTHER  WORDS,  IS  TF.E  BENEFIT  OF  THE  DATA 
WUKTh  THE  COST  Jr  ME  ASL'k  1  N  j? 

THESE  ARE  thE  QUESTIONS  WE  WILL  CONSIDER.  IF  AN  INDICATOR  IS  JUDGED  AS  NOT  IIUOkTANT  OR 
NOT  FEASIBLE,  Ti-Zb  IT  WILL  3E  DElETED  M  THE  INDICATOR  SET.  ARE  YOU  READY  TO 
PROCEED?  O.K .  LOCK  AT  KfiA  NO.  1,  INDICATOR  NUMBER  1.  LC  YC.  CURRENTLY  TRACK  THIS 
INDICATOR  IN  THIS  rJAM?  (The  facilitator  proceeds  in  this  fashion  until  all  indicators 
have  teen  considered.) 


3.5.4  Documenting  Results 


Following  the  review  session,  the  facilitator  prepares  a  rc-port  cf  results.  The  report  >*. 
given  to  the  commander  of  the  target  unit.  The  facilitator  should  strerg'a  recommend  that 
Information  ir  the  report  be  shared-  by  the  commander  with  all  MGEEM  participants,  along  with,  a 
statement  of  the  commander's  pian  to  -use  tne  indicators.  Tablt  1?  suggests  u  format  fer  this 

report. 


Table  12.  Format  for  Documenting  Results  of  Indicator  Review 
Session  -  Base  Weather  Detachment4 


KRA  1:  Mobility  (Wartime  Role);  Be  prepared  tc  deploy  with  customers 

(SAC  and  MAC)  to  provide  weather  support,  forecasting,  and  observing 
to  flight  crews  and  commander^  at  operating  locations. 

Five  indicators,  as  follows: 


Number  of  components  of  mobility  kit  on  hand  (e.g., 

(a)  climatological,  geographical ,  equipment)  Tabulate  from 

Number  of  components  required  for  mobility  Mobility  Checklist 


(b)  Number  cf  kits  per  tasked  area 


Tabulate  from 
Plans  and  Kits 


(c)  Number  of  items  serviceable  (per  kit) 
Number  of  i  terns  in  kit 


Tabulate  from 
Plans  and  Kits 


'lumber  of  people  who  respond  on  time  with  all 

(d)  personal  gear _ Log  Required 

Total  number  of  people  on  mobility 

Number  of  observations  using  TAOIST  equipment  that 

(e)  verify  with,  observations  taken  with  standard  equipment  Net  Available 

Number  of  observations  with  TACMET  equipment 


KRA  2:  Observing  weather  events:  Accurate  and  timely  observation, 
recording,  and  dissemination  of  notices  of  weather  events. 


Four  indicators,  as  follows: 

Number  of  observations  by  observer  that  agree  with 

(a)  observations  taken  by  station  chief 
Number  of  observations  compared 

(b)  Number  of  correct  entries  on  Form  10 
Total  number  of  entries 

Scores  on  tests  of  observing  principles,  determining 

(c)  proficiency,  ana  familiarization  with  manuals  arc 
operating  procedures  and  equipment 

(d)  Number  of  local  observations  taken  (electro-wri ter) 
Number  of  local  observations  required 


Not  Available 

Computer  Report 
Not  Available 


Tabulate  from 
£ lectro- 
'.iriter  Sheet 


etc . 


a Sou rc ps  of  data  to  form  indicators  ..hown  in  rig-.t  column. 
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3.6  Pitfalls  to  Avoid 


Experience  in  applying  the  KoEEM  suggests  the  potential  for  a  number  of  problems  and  mistakes 
to  occur.  If  anticipated,  however,  they  can  be  avoided.  This  section  Identifies  these  potential 
problems  and  mistakes  and  suggests  ways  to  avoid  them  or  deal  with  them. 


3.6.1  degree  of  facilitator  Influence 


The  MGEEM  is  a  participative  methodology  in  which  the  target  organization  defines  KRAs  and 
Indicators  that  Its  members  believe  to  be  important.  It  is  essential  to  remember  that 
participation  Increases  greatly  the  probability  of  acceptance  and  use  of  the  resulting 
productivity  measurement  system  by  the  target  organization.  It  is  therefore  necessary  that  the 
facilitator  not  exert  his/her  will  on  the  group  since  that  tends  to  reduce  the  participants' 
sense  of  ownership  of  the  KRAs  and  Indicators. 

The  role  of  the  facilitator  is  to  use  the  MGEEM  process  to  guide  organizational  participants 
toward  consensus  and  high  quality  results.  The  facilitator  plays  that  role  best  by  ensuring  that 
teams  follow  the  prescribed  procedures  closely.  With  respect  to  the  quality  of  the  content,  the 
facilitator's  role  is  to  raise  questions  about  KRAs  and  indicators  that  may  be  weak.  For 
example,  a  facilitator's  comment  with  respect  to  an  irdicator  could  be  as  follows: 

WHAT  ARE  WE  REAUi-Y  MEASURING  HERE?  or  WHAT  ARE  WE  COUNTING  IN  THIS  INDICATOR? 

Such  a  remark  could  be  helpful  in  guiding  the  group  to  look  critically  at  an  indicator.  On 
the  other  hand,  consider  the  following  comment: 

THIS  INDICATOR  IS  REALLY  WEAK;  1  THINK  IT  SHOULD  SAY  NUMBER  OF  HOURS  WORKED. 

Such  a  rer.ark  by  the  facilitator  could  be  counterproduc  ti  ve  even  though  it  might  be  true.  By 
imposing  his/her  opinion,  the  facilitator  will  cause  the  process  to  break  down  and  will  lose 
rapport  with  the  team. 


3.6.2  Handling  Olsruptive  Team  Members 

The  facilitator  must  build  and  maintain  an  atmosphere  conducive  to  a  free  and  open  exchange 
of  ideus;  however,  discipline  must  be  maintained.  Generally,  the  facilitator  should  confront 
disruptions  in  a  courteous  but  direct  manner.  For  example,  If  a  side  conversation  is  going  cn 
during  the  silent  generation  of  answers  process,  the  facilitator  could  say: 

WE  HAVE  ONLY  ABOUT  5  MINUTES  LEFT,  SO  PLEASE  ALLOW  THOSE  WHO  ARE  STILL  WORKING  TO 
WORK  WITHOUT  INTERRUPTION. 

If  conversation  continues,  the  facilitator  should  speak  to  the  offenders  Indi vldual 1 y ,  asking 
them  to  refrain  from  disturbing  the  group. 

A  potentially  troublesome  issue  is  smoking  during  the  sessions.  To  solve  this  problem,  the 
facilitator  can  raise  this  Issue  with  the  group  at  the  beginning  of  the  session  and  seek 
consensus  regarding  smoking  during  the  session.  If  the  group  is  split  on  the  issue,  then  the 
facilitator  should  say  with  a  sense  of  humor,  not  in  a  confrontational  manner: 

SINCE  WE  CAN'T  COME  TO  ANY  AGREEMENT  ON  THIS  ISSUE ,  PART  OF  MY  JOb  IS  TO  PLAY  "BAD 
GUY."  THEREFORE,  I  WOULD  -REFER  THAT  THOSE  OF  YOU  WHO  SMOKE  CONFiNE  YOUR  SMOKING 
TO  THE  BREAKS.  LET  ME  KNOW  IF  THAT  CREATES  ANY  MAJOR  PROBLEMS. 


Such  a  comment,  made  politely,  will  normally  put  the  issue  to  rest. 


3.6.3  Creating  the  Proper  Mental  Set 


Because  the  MGEEM  is  a  consensus-seeking  process,  the  facilitator  should  promote  commonality 
among  viewpoints  In  every  possible  way.  One  critical  step  in  this  direction  is  to  make  sure  that 
everyone,  throughout  the  process,  approaches  their  task  as  though  they  were  the  commander  of  the 
target  organization,  if  participants  are  allowed  to  maintain  their  own  unique  perspectives, 
particularly  in  the  KRA  process,  this  will  create  difficulties.  For  example,  subordinate 
managers  may  be  motivated  to  justify  their  existence  in  front  of  their  boss  by  listing  everything 
their  people  do.  This  will  extend  the  process  unnecessarily  by  focusing  the  group  on  items  that 
are  too  "micro"  in  terms  of  organizational  level.  To  avoid  such  problems,  the  facilitator  must 
ask  everyone  to  "wear  the  commander's  hat"  throughout  the  exercise  and  look  at  the  "big  picture," 


3.6.4  Not  Allowing  the  Discussion  to  Bog  Down 

A  primary  job  of  the  facilitator  is  to  keep  the  process  moving,  to  stt.  ar;o  maintain  the 
pace.  Normally,  the  problem  here  is  that  the  group  will  get  "hung-up"  on  some  issue  and  spend 
too  much  time  on  one  xRA  or  indicator.  Frequently,  this  degenerates  into  3  discussion  between 
one  or  two  team  members  wno  have  special  knowledge  or  interest  In  the  item,  When  this  happens, 
other  team  members  become  bored  and  want  to  move  on.  The  facilitator  must  be  able  to  "read  the 
group"  and  determine  when  thfs  is  happening.  A  few  skillfully  worded  comments  will  usually 
correct  this  problem.  For  example,  if  two  people  are  emotionally  debating  a  tangential  issue, 
the  facilitator  might  say: 

THIS  seems  LIKE  AN  ISSUE  that  will  be  settled  by  oik  voting  process,  can  we  move  cn? 

On  the  ether  hand,  suppose  the  group  as  a  wheie  is  debating  an  issue  that  the  facilitator 
believes  is  so  important  that  it  could  prevent  the  group  from  coming  to  consensus.  Then  a 
“prompting"  designed  tc  catalyze  the  discussion  is  appropriate.  For  instance,  the  facilitator 
could  say: 

YOU  HAVE  MADE  A  NUMBER  OF  POINTS;  i.ET  ME  TRY  TO  SUMMARIZE  THE  DISCUSSION  AND  FRAME 

THIS  ISSUE.  (State  the  issue  as  you  perceive  it.)  IS  this  THE  QUESTION  BEFORE  US? 

If  the  facilitator  perceives  that  the  group  is  spending  too  much  time  on  j  subject,  then  ne/$ie 
can  call  on  someone  who  has  been  quiet  during  the  debate  and  say: 

JOE,  YOU  HAVEN'T  SAIC  MUCH.  WHAT  00  YOU  THINK  ABOUT  THIS? 

The  discussion  can  bog  down  in  many  ways.  The  facilitator  must  recognize  that  it  is  happening, 
try  to  determine  why  it  is  happening,  and  make  a  decision  either  tc  intervene  in  the  discussion 
or  to  force  the  group  to  move  on. 


4.  USE  OF  PRODUCTIVITY  MEASURES  AS  A  MANAGEMENT  TOOL 

This  section  discusses  procedures  for  using  the  productivity  measures  which  result  from  the 
MGEEM  as  a  tool  for  improving  organizational  productivity.  The  facilitator's  role  is  to  help 
the  organization  develop  and  implement  a  system  for  tracking  and  reporting  productivity 
measurement  data,  but  not  to  make  the  process  fact  1 1 tator-dependent.  In  other  words,  the  system 
must  be  simple  enough  that  the  organization  can  operate  and  maintain  it  without  facilitate 
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assistance.  However,  as  in  Implementing  any  new  procedure,  there  is  an  initial  "hump1'  to  be 
overcome.  The  facilitator  must  help  the  organization  overcome  Initial  startup  inertia  mj 
prepare  the  organization  to  assume  ownership  of  the  system. 

The  key  elements  of  this  role  are  operational  izing  the  indicators,  developing  a  method  and 
format  for  reporting  measurement  data,  instructing  management  with  respect  to  considerations  In 
providing  feedback,  and  suggesting  ways  to  involve  members  of  the  organization  in  the  use  of  the 
data  as  input  to  group  problem-solving  efforts  (e.g..  quality  circles). 


4.1  Obtaining  Data  from  Formal  and  Informal  Sources 


Following  their  development  and  screening,  indicators  must  be  made  operational  by  locating 
and  accessing  existing  data  or  developing  new  data  sources.  Based  on  past  experience  with  the 
MG£FM,  it  is  likely  that  701  to  90 1  of  the  indicators  generated  by  a  target  organization  will 
make  use  of  existing  data.  These  data  may  be  in  the  existing  information  system  or  may  exist  ir 
informal  logs  or  reports  maintained  by  the  organization  for  its  own  use.  Tre  remaining  lot  to 
30t  of  the  indicators  will  require  new  data-capturing  procedures.  This  section  focuses  on 
guidelines  for  capturing  data  which  already  exist.  Oevelcpment  of  new  data  collection  procedures 
is  beyond  the  scope  of  this  discussion. 


4.1.1  Obtaining  Data  from  Formal  Oata  Sources 


Air  Force  organi zatiens  routinely  track  and  monitor  a  wide  range  of  performance  and  financial 
indicators.  In  most  cases,  decisions  about  what  should  be  tracked  are  made  by  higher  head¬ 
quarters,  often  with  field  input,  for  management  control  purposes.  Higher  headquarters  personnel 
design  reporting  procedures,  forms,  and  computer  software  to  input  data,  analyze  data,  and 
generate  management  reports.  The  data  neeced  for  management  control  purposes  may  be  different 
from  the  data  needed  by  a  local  commander  for  Improvement  purposes.  The  important  Implication  of 
this  difference  is  that  data  required  by  a  local  commander  may  exist  in  an  automated  system  put 
may  not  be  reported  In  the  form  that  the  unit  commander  needs.  Thus,  there  are  two  sets  of 
problems  In  obtaining  data  for  the  MGEEM  from  formal  sources.  One  Involves  obtaining  data  from 
management  information  system  products,  and  the  other  involves  accessing  data  which  may  be  input 
to  a  standard  information  system  but  which  uo  not  appear  as  standard  output  products. 

In  the  first  case,  the  facilitator's  main  concern  is  with  the  timeliness  of  the  data.  For 
example,  If  the  Indicator  is  "number  of  late  report  submi ssior.s/number  o<  reports  submitted," 
this  data  element  may  not  be  available  from  headquarters  until  at  least  a  nonth  later.  That  Is, 
the  oata  for  September  may  not  be  returned  to  the  local  unit  until  October,  meaning  that  the  unit 
would  get  feedback  only  after  30  or  more  days'  delay.  On  the  other  hand,  the  local  unit  could 
keep  Its  own  record  of  the  numner  of  reports  submitted  and  the  number  late,  ana  have  these  data 
within  a  few  days  after  the  end  of  the  reporting  period.  The  obvious  tradeoff  involved  is  the 
cost  cf  keeping  additional  records  versus  the  benefit  of  having  timely  data.  The  facilitator's 
role  is  to  present  the  commander  with  this  issue  and  point  out  the  consequences  cf  both  courses 
of  action.  For  example,  the  facilitator  may  have  determined  that  it  would  take  b  hours  of 
clerical  work  to  complete  the  effort  at  the  unit  level  versus  30  minutes  to  take  the  data  off  the 
standard  report  wher  It  Is  received  In  30  to  40  days.  The  loca'iy  generated  data  will  have 
considerably  more  value  as  a  management  and  motivational  too1  than  data  that  a--e  30  to  40  days 
old.  It  is  up  to  the  commander  to  choose  to  generate  the  data  locally  or  wait  ror  the  standard 
information  system  product. 

When  data  are  captured  and  reported  to  nigrer  headquarters ,  out  not  reported  in  the  form 
needed  to  create  the  local  indicator,  a  different  ,i>t  of  issues  arise.  For  instance,  assume  that 
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the  indicator  is  "average  number  of  days  for  supply  unit  X  to  process  ar,  Article  15/average 
number  of  days  for  all  supply  units  in  the  MAJCOM  to  process  an  Article  15."  The  local  unit  can 
track  on  its  own  the  numerator  of  this  indicator.  However,  the  denominator  requires  data  from  a 
higher  headquarters  database.  In  order  for  the  local  unit  to  use  this  Indicator,  it  will  be 
necessary  to  obtain  a  special  information  system  product  that  excerpts  this  Information  from  the 
MAJCOM  database.  Depending  on  the  unit  commander's  resourcefulness  and  "clout,"  this  may  or  may 
not  be  feasible.  Thus,  the  local  commander  Is  presented  with  a  different  set  of  choices.  One 
choice  Is  to  attempt  to  have  the  data  supplied  by  higher  headquarters  and  accept  the  delay  while 
a  reporting  system  Is  modified.  However,  once  modification  is  made  to  provide  these  data  on  a 
monthly  basis,  the  data  received  in  September  may  represent  performance  in  July  rather  than 
August  due  to  reporting  delays.  Therefore,  after  the  effort  has  been  made  to  get  the  data,  the 
Information  may  have  limited  utility  because  of  its  age. 

The  other  choice  Is  to  modify  the  indicator  so  that  it  can  be  developed  locally.  For 
example,  the  indicator  could  be  restated  as  "average  number  of  days  to  process  Article  15s  this 
month/average  number  days  to  process  Article  15s  in  the  past  6  months,"  In  this  way,  the  unit  Is 
comparing  its  performance,  not  with  other  units  in  the  command,  but  with  its  own  past 
performance.  Although  this  is  different  information,  it  is  more  controllable  from  the  unit 
commander's  viewpoint.  The  negative  aspect  is  the  effort  required  to  generate  the  numbers 
locally.  Again,  the  facilitator  should  discuss  these  tradeoffs  wiin  the  local  unit  commander. 


4.1.2  Obtaining  Data  from  informal  Sources 

Informal  sources  consist  of  data  that  are  recorded  on  logs,  locally  developed  forms,  or  othc 
tracking  systems  that  tne  local  unit  may  have  devised,  frequently,  such  rccorcs  are  maintained 
by  the  commander  for  protection  against  customer  complaints,  to  document  perf ort.nnce  problems  of 
the  unit  or  Individuals,  etc.  In  some  cases,  records  are  maintained  to  assist  the  unit  In 
Improving  perf ormance,  In  any  case,  more  information  is  usually  captured  than  is  tabulated  and 
displayed  In  quantitative  form  or  fed  back  to  members  of  the  organization.  The  problems  facing 
the  organization  In  this  situation  are  specifying  the  can  elements  neede.  for  the  indicator, 
defining  the  coding  procedures  required  to  go  from  a  log  to  a  quanti  tat  1 vc  indicator,  and 
allocating  the  resources  to  do  the  work.  However,  the  advantage  Is  that  the  local  unit  has 
complete  control  over  the  process. 


4.1.3  Precise  Specification  of  the  Data-lndicator  Relationship 

Regardless  Of  the  source  of  data  (i.e.,  formal  or  informal),  it  is  r.ecessarg  to  define 
precisely  the  relationship  between  data  elements  In  information  systems  or  Informal  records  and 
each  indicator.  The  specification  must  contain  the  rule;  for  translating  data  elements  Into 
indicators,  tne  time  periods  Involved,  and  the  interpretation  of  the  resulting  indicator.  Tne 
purpose  of  this  activity  is  to  specify  the  relationship  unambiguously  so  that  the  data  recording 
process  car,  be  performed  consistently  over  time.  Otherwise,  the  reliability  and  validity  of  the 
indicators  will  be  seriously  Impaired.  An  example  illustrating  the  correct  use  of  this  process 
Is  shown  In  Table  13. 

For  each  Indicator,  the  facilitator,  with  the  assistance  of  appropriate  unit  personnel,  must 
carefully  specify  the  data-lndicator  relationships  as  illustrated  In  Table  13.  In  addition,  tne 
facilitator  must  assist  the  organization  In  capturing  the  data  elements  required  to  establish  the 
baseline  performance  level  for  each  Indicator,  fne  procedures  required  for  each  situation  will 
vary,  Thus,  this  discussion  cannot  describe  these  procedures  In  detail. 
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Table  13.  Example  of  Oata-Indlcator  Specification 


KSA  no.  1.  Effectiveness,  Legality,  and  Appropriateness  of  the  Unit  Reporting 
Process 

1.  Indicator:  Most  recent  Inspection  rating 

2.  Specifications: 

(a)  Data  source  -  most  recent  published  IG  report 

(b)  Numerical  index  computed  as  follows: 

6  •  Outstanding 
5  •  Excellent 
4  *  Satisfactory 
2  *  Marginal 
0  •  Unsatl sfactory 

(c)  Time  Period  -  Indicator  changes  with  next  IG  or  ME;  Inspection. 

(d)  Interpretation  -  The  higher  the  number,  the  better  tne  performance. 


4.2  Reporting  the  Data 

To  use  productivity  measurement  data  as  a  performance  Improvement  tool,  a  suitable  reporting 
format  must  be  selected.  This  section  describes  two  such  formats.  The  first  is  the  Objectives 
Matrix  (Felix  &  Riggs,  1983),  and  the  second  Is  line  graphs.  The  Objectives  Matrix  provides  a 
means  of  reporting  changes  In  (a)  Individual  performance  measures  and  (b)  total  organizational 
performance  based  on  the  sum  of  all  individual  performance  measures.  The  line  graph  Is  by  far 
the  simpler  of  the  two  methods,  but  depicts  only  changes  In  Individual  performance  measures.  The 
two  methods  can  be  used  separately  or  In  combination.  For  example,  line  graphs  can  be  developed 
for  each  Individual  performance  measure  and  the  Objectives  Matrix  can  be  usee  to  create  an 
overall  performance  Index,  Another  format  for  using  productivity  measurement  data  as  a 
management  tool,  the  product-to-cortl ngencl es  technique,  Is  under  preparation  by  the  Air  Force 
Human  Resources  Laboratory  (AFhRl). 


4.2.1  Developing  the  Objectives  Matrix 

This  section  describes  the  steps  In  constructing  an  Objectives  Matrix  arid  explains  how  to  use 
It.  Tne  example  In  Figure  7  Illustrates  the  use  of  an  Objectives  Matrix  with  one  kRA  which  has 
five  Indicators.  If  an  organization  had  generated  five  KRAs,  each  with  a  number  of  indicators, 
It  would  be  necessary  to  construct  five  separate  matrices  similar  to  the  one  shown  In  Figure  7. 
However,  If  the  commander  concludes  that  constructing  five  matrices  requires  too  much  effort, 
he/she  may  decide  to  select  one  Indicator  for  each  of  the  five  KRAs  and  construct  only  one 
matrix.  In  such  a  case,  line  2  In  Figure  7  would  be  KRA  titles  Instead  of  Indicator  titles. 
Line  3  would  be  five  Indicators,  one  for  each  of  the  five  KRAs.  The  Objectives  Matrix  is  a 
flexible  reporting  procedure  which  can  be  expanded  or  contracted  merely  by  adding  or  subtracting 
columns  as  required  to  fit  the  number  of  indicators. 
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4. 2. 1.1  List  Indlcator/KRA  Titles.  The  first  step  In  using  the  Objectives  Matrix  Is  to  fill 
In  the  Indicator  titles.  This  Is  line  2  In  Figure  7.  The  KRA  and  the  indicators  would,  of 
course,  be  products  of  an  MGEEM  process. 

4. 2. 1.2  Define  Indicators.  Next,  the  formulas  used  to  create  the  Indicators  are  presented 
on  line  3. 

4. 2. 1.3  .Define  Performance  Levels  for  indicators.  The  next  step  Is  to  define  performance 
levels  for  each. Indicator  (Line  5).  This  step  provides  a  means  of  transforming  measures  on  the 
indicators  to  a  common  scale.  The  common  performance  scale  Is  shown  on  the  extreme  right  side  of 
the  matrix  on  line  5.  The  scale  ranges  from  0  (the  worst  possible)  to  10  (1-  to  2-year  goal). 
Corresponding  performance  scales  for  each  Indicator  arc  entered  on  line  5.  These  scales  are 
developed  as  follows: 

1.  Define  the  baseline  for  each  Indicator.  If  an  Indicator,  as  measured  cy  the  formula  In 

Its  column  on  line  3,  Is  already  in  use  by  the  unit,  the  baseline  performance  level  can  be 
established  from  historical  data.  As  a  rule-of-thumb,  3  months  of  historical  data  should 
suffice.  However,  If  there  are  strong  cyclical  or  seasonal  variations  In  the  performance 
measured  by  the  Indicator,  then  data  over  a  longer  time  period  may  be  required  to  establish  a 
stable  baseline.  On  the  other  hand.  If  the  performance  measured  by  the  Indicator  varies 
relatively  little,  then  1  to  2  months  of  data  may  be  sufficient.  The  baseline  normally  Is  the 
arithmetic  mean  (average)  of  the  3  months'  historical  performance.  If  no  historical  data  are 
available,  then  the  Indicator  must  be  tracked  for  some  period  of  time  (e.g.,  1  to  3  months)  In 
order  to  establish  the  baseline.  Once  the  baseline  for  an  Indicator  is  determined,  this  number 
Is  written  on  the  Objectives  Matrix  under  the  column  for  the  appropriate  indicator  and  at  the 
appropriate  performance  level  (line  5)  corresponding  to  a  certain  point  on  the  C  (worst)  to  lo 
(1-  to  2-year  goal)  common  performance  scale.  If  one  assumes  that  0  on  the  0  to  10  scale 

represents  the  lowest  conceivable  level  of  performance  for  this  unit  and  10  represents  a  goal 
aspired  to  In  1  to  2  years,  then  current  performance  Is  the  baseline.  The  rule-of-thumb  Is  to 
establish  current  performance  at  Level  3  on  the  common  performance  scale.  For  Instance,  If  .35 
Is  the  baseline  for  Indicator  1,  Abort  (computed  by  number  of  takeoff  aborts/number  takeoffs),  it 
would  be  written  under  column  1  on  line  5  to  correspond  to  3  (baseline)  on  the  scale.  That  Is, 

the  ,35  would  be  written  In  column  1  on  the  same  row  as  3  (baseline)  on  the  0  to  10  common 

performance  scale  (see  Figure  7). 

2.  Establish  performance  levels  for  each  Indicator  to  correspond  to  0  and  10  on  the  common 

performance  scale.  After  baselines  are  set  for  each  Indicator,  the  next  step  Is  to  define 
performance  levels  0  and  10  for  each  Indicator.  Level  0  is  the  level  of  performance  envisioned 
as  the  worst  that  reasonably  might  occur  within  this  unit  on  the  Indicator  In  question. 

Establish  the  number  and  write  It  In  the  appropriate  indicator  column  on  the  same  line  as  0  in 
the  performance  scale  at  the  extreme  right.  For  Instance,  In  Figure  7  this  value  Is  "Over  .75" 
for  Indicator  1,  Abort,  so  Over  ,75  appears  on  the  bottom  row  on  line  5  to  correspond  with  0 
(worst)  on  the  0  to  10  common  performance  scale.  Repeat  this  step  for  Level  10.  Level  10  is  a 
challenging  goal  that  could  be  reached  if  Improvement  were  continued  for  1  to  2  years.  The 
selected  time  period  should  correspond  to  the  planning  horizon  of  the  organization.  Write  this 
number  In  the  appropriate  Indicator  column  on  the  same  line  as  10  In  the  performance  scale  at  the 
right.  For  Instance,  In  Figure  7  this  value  is  ".01"  for  Indicator  1,  Abort,  so  .01  appears  on 
the  row  on  line  5  to  correspond  with  10  (1-  to  2-year  goal)  on  the  0  to  10  common  performarce 
seal e. 

3.  Fill  In  the  remaining  values.  Now  that  Level  0,  baseline  Level  3,  and  Level  10  have  been 
determined  for  each  Indicator,  the  remaining  Indicator  performance  levels  can  be  added.  The 
simplest  way  to  do  this  Is  to  establish  equal  Intervals  between  the  baseline  and  Level  10  and 
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between  the  baseline  and  Level  0.  This  assumes  that  it  is  equally  difficult  to  move  up  the  scale 
from  one  level  to  the  next.  If  it  is  believed  that  moving  up  the  scale  gets  progressively  more 
difficult  as  one  moves  to  the  top,  the  increments  can  be  made  larger  at  lower  levels  and 
progressively  smaller  nearer  the  top  of  the  scale.  The  performance  scale  for  Indicator  1,  Abort, 
In  Figure  7  illustrates  a  scale  with  unequal  Intervals. 

4. 2. 1.4  Establish  Manager's  Weights.  The  next  step  involves  establishing  weignts  for 
Individual  indicators  (see  Figure  7,  line  7,  labelled  "Manager’s  weight”).  lr  establishing  an 
overall  score  (the  sum  of  performance  levels  on  all  Indicators)  for  a  KRA,  it  is  likely  that 
indicators  are  not  equally  important  in  terms  of  their  individual  contributions  to  mission 
accomplishment.  Therefore,  Indicators  can  be  differentially  weighted  to  reflect  their  varying 
Importance.  This  weighting  process  is  accomplished  by  dividing  100  points  among  the  Indicators. 
In  Figure  7,  100  points  are  allocated  on  line  7  to  the  five  indicators.  Indicator  1,  Abort, 
received  30  points;  Indicator  2,  Delay,  received  20  points;  Indicator  3,  N.Q.R.S.,  received  20 
points;  and  so  on.  This  distribution  can  be  made  by  the  unit  commander,  a  manager,  or  a  group. 
The  group  would  Involve  the  entire  management  team  and  has  the  benefit  of  assuring  a  similar 
understanding  among  team  members  of  the  relative  priorities  of  Indicators. 


4.2.2  Using  the  Objectives  Matrix 

Once  the  above  steps  are  completed,  the  Objectives  Matrix  Is  ready  for  use.  Before  Its  use, 
however,  the  period  of  performance  tracking  must  be  defined.  Normally  this  will  be  monthly.  In 
any  case,  using  the  Objectives  Matrix  Involves  the  following  steps. 

4. 2. 2.1  Obtain  Measurements  for  the  Current  Period.  Measurements  of  performance  on  each 
Indicator  for  the  current  month  are  written  on  line  4,  labelled  "Measurements  (current  period)." 
In  Figure  7,  for  the  month  under  consideration,  the  result  of  the  computation  of  the  formula 
shown  In  line  3  for  Indicator  1,  Abort,  was  .03.  This  value  was  placed  under  Abort  on  line  4. 

4. 2. 2. 2  Obtain  Current  Period  Score.  The  current  period  measurement  of  a  given  Indicator 
(shown  on  line  4)  Is  located  on  the  performance  scale  (shown  on  line  5)  and  then  compared  with 
the  0  to  10  common  performance  scale  at  the  extreme  right.  In  Figure  7,  the  current  period 
measurement  for  Aborts  of  .03  on  line  4  Is  located  on  line  5  and  corresponds  to  Level  8  on  the  0 
to  10  common  performance  scale  at  the  extreme  right.  Therefore,  an  8  Is  written  on  line  6, 
labelled  "Current  period  equivalent  score,”  in  the  Abort  column.  For  the  second  Indicator, 
Delay,  the  current  period  measure,  18  hours,  shown  on  line  4  Is  located  on  the  performance  scale 
(line  5).  The  performance  level  of  18  for  Delay  translates  as  Level  5  or,  the  0  to  10  conroon 
performance  scale  at  the  extreme  right.  Therefore,  a  5  Is  written  on  line  6,  "Current  period 
equivalent  score,"  In  the  Delay  column.  This  process  Is  repeated  for  all  indicators  to  fill  out 
all  values  In  row  6.  It  should  be  emphasized  that  each  higher  level  of  performance  to  be 
achieved  should  be  treated  as  a  hurdle. 

4. 2. 2. 3  Obtain  Weighted  Score.  The  "Current  period  equivalent  score,"  line  6,  Is  then 
multiplied  by  the  value  In  line  7,  “Manager's  weight,"  to  obtain  the  "Equivalent  score  x  weight” 
value  shown  on  the  bottom  line,  line  8,  as  shown  in  Figure  7.  For  Instance,  the  "Current  period 
equivalent  score"  for  Aborts  on  line  6  (which  Is  8)  multiplied  by  the  corresponding  "Manager's 
weight"  of  30  on  line  7  yields  an  "Equivalent  score  x  weight"  value  of  240  on  line  8.  Finally, 
the  overall  score  for  this  KRA,  740,  Is  obtained  by  summing  the  "Equivalent  score  x  weight” 
values  for  all  Indicators  (all  values  on  line  8).  This  sum,  740,  (240+100040+160+100)  Is  the 
total  KRA  or  organl zatlonal  performance  score  for  the  current  period. 

4. 2. 2. 4  Use  the  KRA  Score  to  Assess  Performance  Improvement.  Absolute  scores  do  not  tell 
unit  commanders  or  unit  members  much  about  their  performance.  It  Is  more  useful  to  compare 
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current  performance  with  some  measure  of  performance  in  the  past.  Typi  tolly,  organizations 
compute  monthly  comparisons  between  present  performance  and  a  baseline.  Such  computation  may  be 
made  as  follows: 

Assuming  a  baseline  performance  score  of  710  and  a  current  period  performance  score  of  740, 
the  current  period  performance  index  would  be  computed  as  follows: 

Performance  Index  *  Current  Period  Performance  Score  -  Saseline 

Performance  Score _ X  100 

Baseline  Performance  Score 

Performance  Index  *  740  -  710  X  100 

710 


4.2 


This  Indicates  that  producti vi ty  increased  4.27  from  tne  baseline  (710)  to  the  current  period 
(740).  Similar  indices  can  be  computed  from  the  previous  month  or  any  other  period  of  interest. 
Normally,  Current  versus  baseline  and  current  vtrsus  previous  month  jre  the  primary  irioices  of 
i  merest. 


4.2.3.  Developing  Line  Graphs 


A  second  method  of  reporting  results  of  productivity  or  performance  measures  is  the  line 
graph.  The  line  graph  provides  a  pictorial  representati on  of  performance  changes  over  time. 
Often,  they  are  plotted  with  respect  to  a  goal  or  tdrget  level  of  per lormantc.  As  illustrated  in 
Figure  8,  the  line  graph  requires  the  definition  of  a  time  period  for  the  horizontal  axis  and 
performance  levels  for  the  vertical  axis.  The  addition  of  a  baseline  and  goal  are  optional,  but 
useful.  Once  constructed,  the  line  graph  requires  little  maintenance  other  than  plotting  da?a 
for  each  month. 


Monthly 
Avg.  No. 
Sq.  Ft. 
Cl eaned 
Per  Paid 
Hour 


3c  sel 1 ne 


Pigureg.  Example  of  line  Graph. 
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4.2.4  Using  Line  Graphs 


Line  graphs  can  be  used  with  the  Objectives  Matrix  to  portray  performance  on  each  Indicator 
and/or  overall  performance.  It  is  not  recommended  that  a  line  graph  be  used  to  portray  a  single 
performance/prodjcti  vi  tj  indicator  because  it  could  focus  the  attention  of  the  organisation  on 
one  aspect  of  performance  to  the  exclusion  of  others.  As  a  consequence,  the  measurement  of 
overall  perf ormance  may  be  distorted.  This  potential  problem  is  illustrated  by  the  line  graph  in 
Figure  8.  Suppcse  this  graph  were  displayed  in  a  maintenance  area  to  provide  feedback  to  the 
maintenance  staff.  It  suggests  that  management  wants  to  increase  the  number  of  square  feet  of 
space  cleaned  per  paid  work  hour.  Obviously,  there  are  many  ways  that  employees  can  make  the 
measure  Increase.  One  way  is  to  reduce  time  spent  providing  quality  service  to  customers  so  that 
more  time  could  be  devoted  to  cleaning.  Therefore,  unless  management  also  provides  a  line  graph 
displaying  quality,  the  use  of  the  graph  in  Figure  3  could  be  harmful.  Displaying  a  graph  sends 
a  clear  signal  to  members  of  the  organization  about  what  performance  is  desired.  Therefore,  the 
commander  must  ensure  that  the  right  message  is  sent. 


4.3  Considerations  In  Providing  Feedback 


As  stated  earlier,  feedback  is  a  powerful  tool  for  improving  performance.  AFHRl  has 
supported  an  extensive  research  program  to  identify  practical  guidelines  fcr  the  use  of  feedbatx 
by  Air  Force  organizations  to  improve  and  sustain  performance  arc  productivity.  This  section 
briefly  summarizes  significant  lessons  learned  from  tnis  research.  For  more  detail,  see 
Pritchard,  Bigby,  Beiting,  Coverdale,  and  Morgan  (1931).  A  manual  on  the  use  of  feedback  m 
operational  Air  Force  environments  is  under  preparation  for  AFnRL. 


4.3.1  What  Should  Be  Fed  Back? 

Measurement  -information  contained  in  a  feedback  report  shculo  cover  all  important  aspects  of 
work.  It  should  contain  individual  performance  measures,  as  well  as  total  performance  based  on 
the  sum  of  individual  performance  measures.  It  should  provide  quantitative  information  on 
performance  fcr  t.he  period,  and  it  should  also  specify  "how  good"  that  level  of  performance  is  in 
a  qualitative  -  ense.  Finally,  as  previously  discussed,  indicator  data  should  oe  confined  to 
information  or  dimensions  of  performance  that  are  under  the  control  of  the  target  organi zation. 


4.3.2  What  Should  Be  the  Source  of  Feedback? 

The  feedback  revert  should  derive  from  the  productivity  measurement  system  developed  by  the 
MGEEM.  It  shOuH  be  produced  In  hard-copy  form  (e.g.,  computer  report)  and  te  perceived  as 
coming  from  the  supervisor  or  unit  commander. 


4.3.3  What  Form  Should  the  Feedback  Take? 

Feedback  reports  for  groups  should  be  publicized.  Tnis  may  be  achieved  by  giving  '-eports  to 
all  organization  members  and/or  by  posting  the  data  in  a  conspicuous  place  where  all  organization 
members  can  frequently  view  the  results,  ih.e  frequency  of  feedback  depends  on  the  nature  of  the 
work  processes  on  which  the  measurement  data  are  based.  A  rule-of-thumb  is  to  give  feedback  as 
frequently  as  the  work  cycle  time  permits.  The  feedback  report  should  indicate  how  tr.c  current 
period  cf  performance  compares  with  u  previous  period  so  that  members  of  the  organization  can  see 
changes.  The  Objectives  Matrix  (Figure  7),  combined  with  Hne  graph  feedback  (Figure  3), 
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satisfies  these  requirements.  Another  procedure  which  satisfies  these  requirements,  called  the 
product-to-contlngenlces  technique,  is  under  preparation  for  AFHRL. 


4.4  Linking  Heasureaent  to  6roup  Problem  Solving 


Many  Air  Force  organizations  involve  organization  members  In  problem-solving  activities,  such 
as  quality  circles,  task  teams,  and  labor-management  committees.  Such  groups  differ  as  to  how 
they  select  the  problems  they  will  address.  For  Instance,  quality  circles  members  are  typically 
free  to  make  that  determination  on  their  own.  The  commander  may  suggest  a  problem,  but  members 
are  free  to  accept  or  reject  the  recommendations  of  the  boss.  In  other  arrangements,  task  teams 
are  formed  to  work  on  a  defined  problem  or  Issue. 

Whether  management  or  teams  select  problems,  it  is  often  helpful  to  have  a  mechanism  that 
provides  quantitative  information  on  performance.  The  M6EEM  fulfills  this  need.  In  the  MGEEM 
process,  the  organization  defines  Its  mission  in  terms  of  KRAs  and  indicators  that  cover  the 
Important  facets  of  performance.  In  those  Instances  where  the  Objectives  Matrix  and/or  line 
graphs  reveal  trends  in  performance  that  may  not  be  in  the  desired  direction,  these  trends  can  be 
Investigated  by  problem-solving  teams.  Once  a  team  has  analyzed  the  problem  and  recommended  and 
Implemented  a  solution,  MGEEM  data  can  be  of  further  use  in  evaluating  the  effectiveness  of  the 
Implementation  of  tne  team's  suggested  Intervention. 

Most  employee  Involvement  efforts  have  three  areas  of  potential  weakness.  First,  persons 
Involved  In  such  efforts  wo'-ry  about  whether  the  problems  they  select  for  study  will  help  the 
organization.  Second,  management  support  for  employee  Involvement  efforts  is  often  difficult  to 
obtain  and  maintain.  Third,  groups  Involved  In  such  efforts  experience  a  strong  need  to 
demonstrate  that  their  efforts  pay  off  for  the  organization.  These  questions  are  effectively 
addressed  by  use  of  the  MGEEM. 


4.5  Pitfalls  to  Avoid 


As  Is  the  case  with  any  performance  improvement  strategy,  there  are  potential  dangers  and 
problems  associated  with  use  of  the  MGEEM.  These  can  be  avoided  through  judicious  use  of  the 
technique.  This  section  draws  from  the  authors'  experience  with  the  MGEEM  and  from  recent 
research  on  goal  setting  (Locke  &  Latham,  1984)  to  address  these  potential  dangers  and  problems. 


4.5.1  Fear  of  Being  Measured 


Individuals  often  express  concerns  about  the  measurement  of  their  performance,  but  these 
concerns  can  be  ameliorated.  One  concern,  fear  of  failure,  can  be  dealt  with  by  encouraging 
groups  to  set  goals  that  are  challenging  but  attainable.  Another  concern  Is  fear  of 

repercussions  from  higher  headquarters.  This  Is  often  grounded  In  a  past  history  of  having  been 
"called  on  the  carpet*  by  a  superior  for  a  performance  problem.  Inspection  report,  customer 

complaint,  etc.  The  facilitator  can  reduce  this  fear  through  the  design  of  the  MGEEM  procedure 
and  the  reporting  process.  The  following  steps  are  involved.  First,  Including  the  higher-level 
cotnnander  In  the  KRA  development  process  will  break  down  barriers  and  Increase  understanding  up 
the  line.  Second,  reports  submitted  up  the  line  should  be  summary  reports  that  do  not  provide 
detailed  Information  on  each  Indicator  tracked.  This  will  reduce  the  potential  for 
"micro-management"  from  above.  For  example,  an  agreement  could  be  struck  with  the  next 
higher-level  commander  that  regular  reports  submitted  will  Include  only  the  aggregate  performance 
Index  generated  from  the  Objectives  Matrix.  This  will  provide  a  degree  of  Insulation  for  the 
lower-level  commander,  but  still  ensure  accountability  for  results. 
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4.5.2  Goals  as  Ceilings 


Goals  are  Intended  to  serve  as  targets  that  encourage  performance  improvement.  However, 
goals  can  operate  as  ceilings  which  discourage  further  effort  once  a  target  is  reached.  The  key 
to  avoiding  ceiling  effects  is  to  establish  in  the  organization  an  "eth'c”  of  continual 
performance  improvement.  That  is,  regardless  of  the  level  of  performance  attained,  there  is  an 
understanding  that  there  is  a  need  to  make  further  Improvements.  However,  unless  this  rationale 
is  supported  by  rewards  and  reinforcers  for  excellent  performance,  the  ethic  will  not  be 
sustained.  In  most  Air  Force  organizations,  there  is  an  ethic  cf  high  performance. 
Nevertheless,  commanders  must  be  continually  alert  to  find  new  ways  to  recognize  and  reward 
members  of  the  organization  for  high  performance  to  sustain  this  ethic.  The  MGEEM  can  be  helpful 
in  sustaining  such  an  ethic  because  it  can  be  used  to  set  up  a  situation  in  which  the 
organization  competes  with  Itself  over  time.  This  is  healthy  if  reinforced  by  the  commander  and 
can  help  overcome  celling  effects  associated  with  goals.  Through  competition,  goals  can  be 
raised.  In  addition,  the  Objectives  Matrix  provides  a  means  of  combining  short-term  goals  with  a 
longer-term  focus  by  having  intermediate  hurdles  along  the  way  to  a  1-  to  2-year  goal.  A  manual 
that  includes  a  menu  of  incentives  and  forms  of  recognition,  together  witn  measures  of  their 
attractiveness  to  enlisted  Air  Force  personnel.  Is  in  preparation  for  AFHRl.  This  document  is 
designed  for  use  Ly  commanders,  managers,  and  supervisors  to  provide  rewards  and  reinforcers  to 
help  sustain  a  performance  improvement  ethic. 


4.5.3  Non-Goal  Measures 


"what  you  measure  is  what  you  get."  Because  this  is  true,  commanders  need  to  ensure  that 
only  Important  facets  of  organi zational  performance  are  measured.  The  MGEEM  guarantees  this  by 
forcing  the  Indicator  development  process  to  take  place  in  relation  to  KRAs.  Therefore,  with  the 
MGEEM,  it  is  impossible  for  the  organization  to  develop  a  set  of  indicators  that  are  easy  to 
measure  but  which  miss  the  most  important  performance  areas. 


4.5.4  Gaming  the  System 

As  with  any  measurement  system  there  is  the  potential  for  “gaming."  People  are  creative  with 
respect  to  "giving  the  boss  what  he/she  wants  to  see."  Controls  for  this  tendency  must  attack 
the  motivation  to  "fake"  rather  than  attempt  to  make  the  system  fakeproof.  That  is,  the 
facilitator  should  concentrate  on  developing  the  participants’  motivation  tc  create  a  good 
system.  Much  of  what  has  been  said  above  deals  with  this  issue.  For  instance,  reducing  the  fear 
of  higher  headquarters  intervention.  Involving  the  organization  in  the  development  of  the 
measurement  system,  and  creating  an  ethic  of  continual  performance  Improvement  are  all  efforts  to 
reduce-  the  motivation  to  "game  the  system."  There  is  no  desire  to  game  a  system  that  members  of 
the  organization  feel  truly  belongs  to  them.  If  the  MGEEM  process  is  properly  carried  out, 
successful  implementation  of  the  system  is  assured. 
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APPENDIX  A:  STRUCTURED  INTERVIEW  FORM 


Format  for  Interview  with  Unit  Commander 


Note  to  Interviewer 

Following  introductions  and  preliminary  discussion  to  establish  rapport,  the  following 
questions  may  be  used  to  gather  information.  These  are  simply  opening  questions  and  should  be 
followed  by  "probes"  to  obtain  more  detail.  Examples  of  probes  are:  "Tell  me  more  about  that." 
"Could  you  elaborate?"  "Could  you  give  me  a  specific  example?" 

The  purpose  of  the  interview  is  to  make  possible  the  construction  of  a  systems  diagram  of  the 
organization.  It  is  helpful  to  visualize  this  diagram  as  the  interview  is  conducted. 


1.  What  Is  the  mission  of  this  unit? 

2.  What  are  the  major  products/services  of  the  unit? 

3.  Who  are  the  principal  customers  of  the  organization? 

4.  What  about  staffing  levels?  How  many  military  and  civilian  personnel  are  authorized? 
Assigned?  What  Is  the  breakdown  by  grade  level  and  experience? 

What  other  units  does  this  organization  depend  upon  for  information  or  support  in  order 
to  get  the  job  done? 

6.  What  degree  of  control  do  you,  as  commander,  have  over: 

number  of  personnel  hours? 
material  and  equipment  acquisition? 
capital  Investment? 
energy  consumption? 

7.  How  Is  this  unit  evaluated  by  higher  headquarters? 

8.  What  primary  Indicators  do  you  use  to  tell  you  that  the  unit  is  doing  what  It  Is  supposed 
to  do? 

9.  Do  you  have  a  standard  In-briefing  for  visitors?  If  so,  could  I  have  a  copy  of  your 
briefing  slides?  What  other  written  material  would  help  me  better  understand  your  mission  and 
organizational  structure? 

10.  What  else  should  I  know? 


precediNQWVGEblank 
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APPENO I  X  B:  MEASUREMENT  IMPLEMENTATION  PLAN  FORMAT 


1.  Purpose  of  Measurement  Effort 


2.  Definition  of  Productivity  for  tnis  Organization 


3.  Specification  of  the  Target  Organization  (What's  included  and  what's  not) 


4.  Force  Field  Analysis  Summary: 
a.  Forces  For: 


b.  Forces  Against: 


5.  Strategies  to: 

a.  Increase  forces  for: 


b.  Decrease  forces  against: 


6.  Implementation  Steps 


Key  Dates 

Step  Start  Revi ew  Person  Responsible 

1.  _  _  _  _ 

2. 


N 


7.  Communication  Activities 

Ac t i  vity  Start  Date  Person  Responsible 

1. 

2. 


H  _ 

8.  Coordination/Liaison  Requirements 
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KRA  1 

1-1. 

1-2. 

1-3. 

1-4. 

1-5. 

1-6. 

1- 7. 

ICRA  2 

2- 1. 

2-2. 

2-3. 

2-4. 

2-5. 


APPENDIX  C:  SAMPLE  kRAs  AND  INDICATORS 
Propulsion  Organization  5 
Key  Result  Areas,  Indicators  and  Data  Sources 
:  Sufficient  number  and  quality  Of  J-E7  engines  to  meet  mission  requirements 


Number  of  operable  spare  engines 
Total  engines  on  hand  in  shop 

Number  of  test  cell  rejects 
Number  of  engines  tested 

Number  of  premature  removals  within  100  hours  (Time  x+1 ) 
Number  of  premature  removals  within  100  hours  (Time  x) 

Number  of  repair  actions  on  wing  within  100  hours 

(Time  x+1 ) _ 

Number  of  repair  actions  on  wing  within  100  hours 
(Time  x) 

Number  of  3-levels  assigned 
Total  branch  manning 

Actual  man-hours  available 
Man-hours  assigned 


Number  of  failed  personnel  and  technical  evaluations 
Number  of  evaluations 


Ensure  morale  and  well-being  of  personnel 


Frequency  people  recognized  (Time  x-»1 } 

Frequency  people  recognized  (Time  x) 

Number  of  defects  detected  on  the  job  (Time  x*l ) 
Number  of  defects  detected  on  the  Job  (Time  x) 


Number  people  late  for  work  (Time  x-1) 
Number  people  late  (Time  x) 

Number  of  disciplinary  actions  (Time  x-»l ) 
Number  of  disciplinary  actions  (Time  x) 


Hours  of  compensation  cime 
Hours  of  overtime 


AP  Form  1534,  engine 
management  system 

AF  Form  1534,  engine 
management  system 

AP  Perm  1534,  engine 
management  system 

Base  level  information 
system  (BUS) 


Local  training  form 
Unit  Manning  Document 

AF 10  Form  349, 

Monthly  Maintenance 
Summa ry 

AF  Form  2419,  Routing 
and  Review  of  Quality 
Control  Reports 


Not  available 


QC  reports  and 
maintenance  standard 
evaluation  team 
(MSET)  report 

Roll  call  books 


Unfavorable  Informa¬ 
tion  file  ( LI  I F )  and 
consolidated  base 
personnel  office 
(  CP0)  personnel  file 

Locally  maintained 
record 
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2-6. 

Number  of  requests 

for  change  of  duty  section  (Time  xrl) 

loca 1 ly 

mal ntained 

Number  of  requests 

for  change  of  duty  section  (Time  x) 

record 

2-7. 

Cy  section,  Namier 

of  counseling  slips  (Time  x * 1  ; 

Local  ly 

maintained 

Number 

of  counseling  slips  (Time  xi 

record 

KRA  3: 

Proper  ass igr.r.ient 

cf  personnel 

3-1. 

By  section,  Number 

of  qualified  people  available 

AF  Form 

623,  On- 

Number 

of  people  assigned 

the-Job 

Record 

Training 

3-2. 

By  section,  percent  of  3-,  5-,  and  7-1evels  assignee 

Locally 

maintained 

record 

3-3. 

By  section.  Total 

number  of  people  assigned 

Locally 

maintained 

Total 

authori  cations 

record 

3-4  By  section,  average  man-hours  worked 

Expected  man-hours  available 


3- 5.  Number  of  personnel  turnovers 

Number  of  personnel  turnovers 

KRA  4:  Safety  of  personnel 

4- 1 .  Number  of  safety-related  Incidents  (Time  x+1 ) 

Number  of  safety-related  Incidents  (Time  x) 

4-2.  Number  of  detected  safety  violations  (Time  x+1) 
Number  of  OSV  (Time  x) 

4-3.  Hours  lost  per  month  because  of  accidents 


4- 4.  Number  of  times  protective  apparel  not  worn  when 

required  (Time  x+1) _ 

Number  of  times  protective  apparel  not  worn  when 
required  (Time  x) 

KRA  5:  Proper  tools  and  test  equipment  and  facility 

5- 1 .  Number  of  appropriate  Q.C.  and  MSET  writeups  (Time  x+1) 

Number  of  appropriate  Q.C.  and  MSET  writeups  (Time  x) 

5-2.  Number  of  tools  on  backorder  (Time  x+1 ) 

Number  of  tools  on  backorder  (Time  x ) 

5-3.  Number  of  proper  tool  self-inspections  (Time  x+1) 

Number  of  proper  tool  self-inspection  (Time  x) 


AFT  Form  349, 
Monthly  Maintenance 
Summary 

Locally  maintained 

recor 


Q.C.  record,  safety 
office  record 

Q.C.  reports 


winy  Safety  office 
record 

Locally  mal ntal ned 
record 


Q.C.  and  MSET  reports 


AF  Form  601,  Equip¬ 
ment  Action  Request 

Locally  maintained 
record 


4B 


5- 4.  Number  of  not  repairable  this  station  ( NRT  S )  o-U  4  (lime  «.+ 1  ) 

Njrcber  of  not  repairable  this  station  (n(U5;  oJe  (Tine  x) 

KRA  6:  Proper  conduct  ana  discipline  of  perso'-'.e’ 

6- 1.  Average  number  of  counseling  sessions  before  iisc i|.l  injry 

action 

6-2.*  Number  of  disciplinary  actions  (Time  *»  1  . 

Number  of  disciplinary  actions  (Time  x) 

6-3.  Number  of  on-base  incidents  (Time  x+ 1 5 


A:'0  i-o rip  349 

■  ocal maintained 
recc-t  i 

'Ir  and  CBPO 
personnel  file 

Ai '  ;  slice  record 


APPENDIX  0:  KRA  DEVELOPMENT  SESSION  BRIEFING  ANO  AGENDA 


0800  Introduction 

Introduction  of  Facilitator  and  Participants 
Purpose  of  the  Measurement  Activity  (Unit  Coemander) 

Overview  of  MGEEM  Process 

Perspectives  on  Defining  Measures  (Slide  1) 

What  are  Key  Result  Areas?  (Slide  2) 

Purpose  of  This  Meeting 
Questl ons 

0830  Productivity  Overview 

Discussion  of  Organization  Diagram  (Slide  3  Is  sample  -  should  use  diagram  for  the  target 
organization) 

Definition  of  Productlv4 ty  for  the  Target  Organization 
questions 

0900  Nominal  Group  Technique  -  Round  1 

Statement  of  NGT  Question 

State  That  Frame  of  Reference  Is  That  of  Unit  Commander 

Silent  Generation  of  Answers 

Round-Robin  Listing 

Discussion  and  Clarification 

Vote  1 

1030  Break  (Results  of  vote  1  are  tallied  during  break) 

1045  NGT  Task  -  Rouid  2 

Olscusslon  of  Noting  Pattern 
Modification  cf  Items 
Vote  2 

1130  Tally  Vote  2  (Adjou-n  If  sufficient  agreement  Is  obtained;  otherwise,  repeat  the  Round  2 
process.  Upor  adjournment,  establish  time  for  followup  "reasonability  check"  meeting). 
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ORGANIZATION  DIAGRAM 


Date 
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APPENDIX  F:  AGENDA  AND  BRIEFING  FOR  INDICATOR  DEVELOPMENT  SESSION 


0800  Introduction  and  Overview 

Purpose  of  the  Measurement  Activity  (Unit  ■_i>*nande>' ) 

Presentation  and  Discussion  of  KRAs  Developed  Ly  Team  A 

Task  for  Team  8 

Questions 

0830  Basic  Considerations  in  Indicator  Development 

Definition  of  Productivity  for  the  Target  Organization 
Oeslrable  Indicator  Characteristics  (Slide  U 
Examples  of  Performance  Indicators  (Slide  ?) 

Examples  of  Measures  (Slide  3) 

Indicator  Development  Quiz  (Appendix  S) 

0900  NGT  Process 

Ground  Rules  for  Nominal  Group  Technique 

Round  1  —  KRA  no.  1  (if  group  decides  to  start  with  KRA  ).  The  first  KRA  selected 
should  be  the  one  that  is  easiest  to  measure). 

1015  Break 

1030  (Continue  until  indicators  have  been  generated  for  all  xRAs.  Typically,  the  process 

takes  60  to  90  minutes  per  KRA) . 
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DESIRABLE  INDICATOR  CHARACTERISTICS 
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PERFORMANCE  INDICATORS 
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PAST  IMPACTS  OR  GOALS 


APPENDIX  G:  INDICATOR  DEVELOPMENT  Ql i Z 


Which  one  of  the  following  phrases  woo'd  be  an  acceptable  output  indicator’:' 

i 

a.  Pull  engines 

b.  Hours  worked 

c.  Number  of  engines  r  led  per  month 

Which  of  the  following  .  cators  measure  possible  efficiency?  (Choose  one  or  more.) 

a.  Number  of  inspections  conducted  this  period 
Number  of  hours  spent  on  inspections 

b.  Number  of  quality  control  errors  detected  this  period 
Number  of  products  produced  this  period 

c.  Actual  expenditures  this  period 
Planned  expenditures  this  period 

d.  Actual  number  of  engines  repaired  per  shift 
Standard  number  of  engines  repaired  per  shift 

e.  Average  tine  between  repairs 

f.  Percent  of  total  work  hours  vehicle  is  available  for  use 

g.  Number  of  dollars  damage  to  base  property  due  to  unforecasted  severe  weather 

h.  First-term  reenlistment  rate  for  unit 
First-term  reenlistment  rate  for  base 

In  item  2,  above,  which  indicators  are  effectiveness  indicators? 

Which  indicator  below  Is  controllable  from  the  point  of  view  of  a  maintenance  branch? 

a.  Number  of  people  authorized 

b.  Average  number  of  months  in  AFSC  per  person  assigned 

c.  Average  number  of  hours  to  overhaul  an  engine 
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Answers  to  Indicator  Development  Quiz 


1.  Alternative  "c"  is  the  answer.  Alternative  "a"  simply  describes  a  task  and,  as  stated,  is 
not  quantifiable.  Alternative  ''o“  does  not  clearly  specify  what  is  to  be  measured;  i.e.,  number 
of  hours  worked  per  employee  per  month. 

2.  Alternatives  "a,"  ”d,"  and  “f"  arc  efficiency  indicators;  "a”  is  an  output/i  nput  ratio;  "d" 
is  a  comparison  between  actual  and  standard  performance;  and  "f"  measures  utilization. 

3.  In  item  2,  alternatives  "b,"  "c,*  "g,1'  end  "h"  are  effectiveness  indicators;  "b"  is  a  measure 
of  quality;  "c“  is  a  measure  of  performance  in  relation  to  budgeted  goal;  "g”  is  a  measure  of  the 
external  Impact  on  the  unit  itself.  Obviously  “h"  is  not  a  pure  measure  of  irternal  impact  since 
reenlistment  rate  Is  affected  by  other  influences  besides  the  way  the  unit  is  managed. 
Nevertheless,  this  Indicator  is  partially  controllable  from  the  unit's  point  of  view,  which  makes 
alternative  ”h"  a  crude,  but  perhaps  acceptable,  indicator. 

4.  Alternative  "c”  is  correct.  The  unit,  through  its  work  activity,  can  impact  the  number  of 
hours  required  to  overhaul  an  engine.  However,  it  has  no  direct  control  over  the  number  of 
people  authorized,  alternative  "a,"  or  the  experience  level  of  the  personnel  assigned, 
alternative  “b.“  These  factors  are  controlled  by  the  Air  Force  manpower  and  assignment  processes 
and  do  not  reflect  the  actions  of  an  indivicual  organi zati on. 


APPENDIX  H:  INDICATOR  DEVELOPMENT  WORKSHEET 


APPENDIX  I:  INDICATOR  REVIEW  GHEE T 
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